I think it’s because they get to witness uncommon procedures play out.

The rest of the population just wants to get on with life.

I suspect it's a bit of insecurity; what would you do if your application's landscape suddenly went offline? In theory it's all infrastructure-as-code and pressing 'start' will boot everything up, pull up the last backups, etc... but when's the last time you tested that? When's the last time you did a training?

I've been working at an energy company for a few years now; while we have an office fire drill a few times a year, there is never an IT outage training. At best a load test late at night. And they paid the price a few years ago; due to the energy crisis, people checked their energy contracts and prices more often, causing 2-5x more load than normal. And the IT hadn't been updated, even though they were warned for a decade that their IT had bottlenecks (single database etc) and they would run into issues sometime.

Well, it basically was, wasn't it, just a fairly orderly one?

https://news.ycombinator.com/item?id=43820964

https://news.ycombinator.com/item?id=43824032

At least HN did better than the BBC that started by reporting and I quote "It is thought to be caused by a Cyberattack".

Also should be noted the Guardian is not doing better as the article does not clarify what the cause was. Most likely explanation at the moment is this:

"What is "Induced Atmospheric Vibration"?" - https://physics.stackexchange.com/questions/848666/what-is-i...

"Diagnosis and Mitigation of Observed Oscillations in IBR-Dominant Power Systems - A PRACTICAL GUIDE" - https://www.esig.energy/wp-content/uploads/2024/10/esig-rpt-...

12:00 I was working when around noon my computer display switched off. I tried turning on a light in the room: no light either. I went to the main circuit breaker, switched it off/on again : still no light. I went out to the street. The house is in a small suburban area so each house has it's own meter. The smart meter was not displaying any light or text. I knew that in my village a blinking red LED on the smart meter means electricity flowing. A fixed-constant red LED light means the grid is on but the house is not consuming electricity. No light and no text means there is no electricity reaching the smart meter. I checked all the neighbors' meters and I saw all lights were off. I started walking to the next street and was checking when another man came out. He told me he was also missing electricity. I checked Google, on the News tab there is no article about "{my_city} outage". So I started to walk towards the center. I called my parents which are in another EU country, they answered and told me all was good and they had electricity. So no electricity but the 3G mobile network was on (it remained on for about 4 hours before both 3G mobile network and normal phone signal went off). I started to walk toward the city center checking meters to understand how big is the problem, all streets are off. I realized my car had only 1/3 of the tank full. I decided to try the petrol station. On the way there, there was a train level crossing, the lights were blinking and the warning sound was on. A young man was keeping the train bar up and gesturing for cars to pass by without risk. I quickly heard him shouting "No electricity". I passed under with the car since many cars were doing it and I knew electricity was off so probably the bars were not working properly.

PETROL STATION

After 30-45 minutes of outage there were 4-5 cars parked around the petrol station pumps. No-one was actively using the pumps to fill the tank. I saw 4-5 people arguing, not sure about what. I went in the petrol station shop and there was an usual long queue of 30 people, the employee at the desk was writing on a paper. I asked another employee if it was possible to put petrol and she said no. I decided it was too late and it was not worth it to wait there. Given also some people already seemed agitated.

SUPERMARKET

Supermarket on the way was closed because it was a bank holiday on Monday. Maybe that's also why the outage had been more likely to happen on that day with less people checking things.

TRAIN LEVEL CROSSING

I was driving back home. At the train level crossing the same guy from before and another man were holding up the bars. I asked them if they needed some tools to rig the bar up and they said yes. Rushed back home and got some tools and wood and ropes. Got back there in 15 minutes. They had found a way to remove the bars by unscrewing a bolt. I chat with them. Another man passe by car and said he had notified the local police they had to remove the bar and the police was aware. At the moment I though those men were smart and capable and thinking on their feet. We exchanged numbers. We promised to all check the electricity at home and come to fix the bar at the train crossing. (I did go to check this morning at 8am when I woke and electricity was back.) The bar was still missing and lights were blinking. There was a long queue of 40 cars. I checked no train was passing and I started to let the people pass by. After I noticed a technician on the track 100 meters out in a yellow suit. I went to speak with him. Walking on tracks and pebbles is not super fast since you can't run easily. The technician said the trains were not running today. I double checked he was aware the protection bars were removed and that he would put them back when he knew the train were passing.

BACK HOME

When I came back home from the train level crossing during the outage I decided to do what I could. I filled up all bottles of waters I had in the house. Even if the outage was temporary I can still drink that water and it wasn't a waste.

STARLINK INTERNET SATELLITE

I have a Starlink antenna that I use in the summer during the holidays. I rushed to resubscribe that I had cancelled. As far as I understand if you don't have internet is not possible to subscribe and use the Starlink (which you need for internet). I also noticed there is a new cheaper plan for 9€ for 10GB in Spain. So I will be permanently subscribed to that to avoid the fear internet goes away and I can't subscribe back to Starlink and make it work.

FIBER OPTIC/DSL

Anyway this time the 3G mobile network was working and I had a small UPS that was keeping the modem and router working. The fiber connection with the modem&router powered by external battery (UPS) was giving normal internet. It worked for 4 hours until around 18:00 when even with power to the modem&router the fiber stopped working. It was around the same time the 3G/mobile signal had stopped.

I'm not sure if there 4 hours diesel generators somewhere or there was some priority line for telecoms. Please let me know if you know anything.

ECOFLOW BATTERY AND SOLAR PANELS

Somehow I had been a bit paranoid in the past few weeks and I had bought 1 month ago an Ecoflow Delta 3 Plus with 1024Wh, 2 solar panels of 800W each and the Powerstream which is a microinverter from Ecoflow to provide up to 800W by plugging in a normal socket of the house. I had decided to actually test them this weeks during the Easter holidays. I had left the Ecoflow battery uncharged because I thought it would be more obvious with the solar panels input. I cursed by perfectionism. Luckily the Ecoflow came 25% pre-charged.

The UPS was down in 40 minutes already and I saw the Ecoflow battery could maintain the modem&router for 4 hours. I was not sure I would be able to set them up correctly if internet went down. Also there were only 2-4 hours left of sun to charge the battery. So I decided to rush it and try to get the solar panel work. I did manage to connect all stuff but I realized I had forgot to buy the XT60i connector to connect the solar panels directly to the battery without the Powerstream. The shadow of a building in front obscured the patio where the solar pannels where around 18:00. Somehow the INPUT of the solar panels even in the last minutes with sun was reported as 0Watt. I went online and double check but I could not find anything obvious. Time was up for the day (or better said, Sun was down for the day). I decided to switch off the battery and save power for the day after with sun to have internet.

BIKING AROUND AND CHINESE SHOP

Given the sun was down and I couldn't do more I decided to take my electric bike and go around. My first petrol car had 1/3 gasoline tank full. I had just bought a second van with popup top roof last week. The tank was in reserve from the dealer and I had planned to go fill it like tomorrow. I cursed myself. Luckily I thought I still had the electric bike to charge from solar. Not sure exactly for what. So I went around the suburban neighborhood, and people were having a good time. It's Eastern holidays, one bar was opened selling drinks. I passed in front of a small Chinese shop. The shop was open with old people and kids going in and out. In Europe there are commonly small shops with tons of useful house things. They are usually managed by a whole Chinese family. Some people were buying torching and batteries. Only cash accepted of course. I already was stocked on torches and batteries and tools like that. But I browser around anyway. Some kids were playing hide and seek in the dark halls of the little shops. It was very cute. Out of the outage on the first hours people were very excited and happy. I bought a 4 in 1 key with triangle/square heads for 2.30€ (3$). It was good to have anyway and when I had checked the smart meters of the neighbors some required a triangle key to be opened and see if they were working. I don't plan to do anything nefarious. It seemed I was among the fastest and most active responders of the situation so it feels ethically okay for me to own this technician key. Just in case.

THE NEWS

Checked internet through Starlink one last time before bed. They said the outage would resolve from 21:00 to 1:00am. Some friends in Valencia around 20:00 CET told the power was back there.

GOING TO SLEEP

I wen to bed. I left the Ecoflow Delta 3 Plus plugged in the sockets in case the electricity came back during the night (and would go away before I would wake up).

LIGHT BACK

Around 1am I was woken up by the bed-side table turning on. Half-asleep I switched it off and kept sleeping.

WAKING UP

This morning electricity was back when I woke up around 8am. As I said in the TRAIN LEVEL CROSSING section I went there to check. I'll spend the day properly learning how the Ecoflow battery and solar panels work together with calm.

Thank you for reading until here!

By just politely asking you can hit tons of different terms in a day.

> REN said: “Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as ‘induced atmospheric vibration’. These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European network.”

- Redes Energéticas Nacionais (REN) is a Portuguese energy sector company

That does not answer the interesting (and slightly conspiratory) question. Did renewables play any negative role in this at all?

https://en.m.wikipedia.org/wiki/Inertial_response

There really is no one-size-fits-all solution, except for having an energy mix of renewables, with existing thermal power on standby.

Everything else will be covered by renewables + battery.

The UK was very very close to a blackout last winter for this exact reason. If Denmark weren't able to suspend maintenance on part of their grid to max out the Viking HVDC (it was running at half capacity) there would have been loss of supply. Potentially contained to one region and temporarily but it's hard to say for sure.

What's worse is the communication from NESO (now government owned, very recently) was downright misleading to how close it came.

Which had already happened a few times across big parts of Europe, with weeks of overcast and low wind weather.

Specifically for Spain, a better backup wil probably be more interconnects with France and their stable nuclear power.

The reality doesn't matter to 90% of people.

Richard Tice, the Reform party’s deputy leader and energy spokesman, said the events in Spain should be a warning to Britain and showed the risks of net zero.

He said: “We need to know the exact causes but this should be seen as a wake-up call to the eco-zealots.

“Power grids need to operate within tight parameters to remain stable. Wind and solar outputs by contrast, vary hugely over long and short periods so they add risk to the system. The UK’s grid operators and our Government should take heed.”

That's the same Richard Tice that blamed a substation fire on Net Zero and wants to tax renewable power to subsidise non-renewables.

A roundup of people jumping to conclusions is at

https://reneweconomy.com.au/spains-blackout-has-already-trig...

> The core priority of grid services right now is getting power restored, rather than establishing the narrow chain of causal events, which will take weeks or likely months. That will be more than enough time for the narrative to be set in stone; immune to whatever reality emerges from careful investigation.

Wait until you find Spain it's the 2nd most mountainous region in Europe. Hint: anything non-coast can have dry summers and cold winters, brutal in the mountain regions such as Asturias and Leon, were they burnt coal like crazy, and having temperatures below 0 was and it's still the norm, not the exception. And not just the south of Cantabria, Asturias and so; the Castilles, Aragon and the mountain regions can amaze tourist as if they were phonied expecting a 'sunny' Spain like the beaches.

Mostly humid and foggy panoramas with mountains; and hot and dry summers in the Castilles; but it's like Ohio and the inner USA: you have extremes in both sides. Scorching sun and crazy cold winters, and yet the corn -wheat in Spain- raises like nothing.

So, one valid approach for one region can be void for another one. You can't except to set a global energy policy that works everywhere. Kinda like the US; not everything it's like Texas or Florida.

And, yes, we had tons of dams too near the Tagus and Duero rivers, they can produce tons of power too.

Nowadays, well, I'm pro nuclear, coal was something obsolete since the 80's and 90's.

Wide scale blackouts aren't exactly unknown, no matter what the generation.

And while you and I may agree, sadly the people who need convincing are those who listen and vote for the parties who say "it doesn't matter what we do here when China builds more coal power plants a week than we've ever had".

When they say "the reason you were without power is solar", and the opposition say "doesn't matter, you can't have gas", they aren't going to vote for the opposition.

No, we do not, unless you use a very narrow definition of sustainable. Nuclear will play a big part, especially in Western Europe, thanks to France which exports to all its neighbours and is a good stabilising provider.

And other than that, who knows. Green hydrogen could be a solution for energy storage.

As of early September 2022, 32 of France's 56 nuclear reactors were shut down due to maintenance or technical problems. In 2022, Europe's driest summer in 500 years had serious consequences for power plant cooling systems, as the drought reduced the amount of river water available for cooling.

And rules around this have been adjusted to have less of an impact for the next "once in a 500 years" event.

Also, in the same way that France's nuclear power generation being out was covered by German and British renewables, during low points of renewable generation in those countries, French nuclear covered the gaps. That's the point of an interconnected network.

My personal hunch is they were struggling to balance the grid as peak solar approached with too much supply. The last snapshot of data showed less than 1GW of CCGT which is not a lot for a grid of this size (the UK runs 2.5GW CCGT 24/7 for inertia and balancing reasons, even if there is excess renewables available - they will be paid to be turned off). Spain does also have hydro which can respond to the grid in fairness, but I am unsure how well prepared they would be to do this if CCGT is the main balancing source.

I also saw data showing Spain was exporting to France heavily at the time. I suspect if that interconnector went down (as rumored), you'd have potentially 1GW+ more load with nowhere for it to go.

This may have then caused the disconnection events (the spanish grid operator said there were 2), as various plants trip as they saw the grid becoming unstable. They recovered from the first but couldn't recover from the 2nd, there isn't a huge amount of detail on these.

This is basically exactly what happened in the 2019 outage in the UK (and from comments on HN yesterday was also a major factor in the Brazilian outage recently) which affected SE England and London (but didn't cause a full grid collapse, I suspect averted because of more CCGT online to balance).

In that event:

1) 400kV cable goes down. Power is rerouted. Was a lightning strike which is pretty common.

2) This caused a very brief change in the voltage/frequency on the immediate substation that was taking power from Hornsea 1 offshore wind farm to the national grid

3) Hornsea 1's voltage detection equipment was too sensitive and tripped, dropping 700MW of supply. It shouldn't have tripped for such a short drop while they rerouted grid power to bypass the out of service 400kV line.

4) This then caused a cascade of other generating assets to trip (also too sensitive), causing a major drop in grid frequency and this then triggered various distribution points in England to disconnect automatically to try and drop load by 5%. Furthermore all 'variable' generation is requested to immediately raise output to balance the system while load shedding goes on.

5) The load shedding operation did not work as expected. They expected drop was 850MW but only 350MW was achieved. This is likely due to embedded solar which they hadn't taken into account in their calculations (or one of the reasons). This causes frequency to drop further and another level of load shedding to occur.

6) Frequency recovers.

Even worse, there was another potential near miss here - the DNO started reconnecting customers before they had the goahead to connect from the grid. It is lucky that another outage didn't occur because of this as the grid was not fully checked and they should have waited.

Anyone saying that renewables are not affecting grid stability are unfortunately just wrong. I am huge proponent of renewables but we have the ratio of solar/wind/batteries/other inertia (flywheels) really wrong. We should not be running so much variable capacity with so little storage.

( https://en.wikipedia.org/wiki/(Ghost)_Riders_in_the_Sky:_A_C... )

Yippie-yi-oh, yippie-yi-aie!