I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent company that is unable to admit a mistake.
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks than to burn it by causing a massive, visible problem.
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
Unfortunately as you said, the range is abysmal since it uses BLE. It’s barely enough for a box store but it’s better than nothing. I like the idea of the lilygo pager too. I first thought it was e-ink from some of the pictures but sadly it was not.
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
This is unrealistic and seems to be biased by some kind of broad un-focused hostility. Yes, maybe they were overstaffed. But it's reasonable to suspect that leadership overcut, given the current climate and the number being 15,000. Your characterization of Twitter predictions relies on cherry-picking and ignores the actual impacts, and there's no evidence that the system goes down without "constant" intervention from "thousands". Your tone also implies that large, complex systems, even if designed well, don't normally require a lot of maintenance from many people.
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?
Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
That's very often the case. I bet there are frantic calls being made right now (probably using a competing carrier!) to Ericsson/etc to come to their rescue, which they happily will in exchange for a hefty fee.
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
It's a population map lacking Dallas for sure ... which might mean something real ... but that doesn't match the per-city charts they offer. Who knows.
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
We had edge delivery issues when I didn't use my ISP's DNS, especially from Apple. Not exactly sure of the mechanism, but downloading Xcode would take 2 hours instead of 10 minutes.
Most users don't notice any real differences vs using their ISP DNS, and seeing it up and configuring it is yet another thing to take time or go wrong.
Many reasons not do use the provided DNS. First, you don't want to give the ISP more information on your browsing habits than it can already gather otherwise. Second, in some countries, ISPs censor websites at the request of of the movie and music industries. Those are enough reasons to rely on a neutral DNS provider like Quad9 or your own DNS server.
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
I didn't pay for any navigation for my car, so I'm not sure what your point is? ISPs provide DNS. People shouldn't have to fuck with the internet's phone book when they plug their modem in (and they haven't for a very long time). Maybe we can expect more from the people who provide the services society relies on, instead of just saying "why don't you just..." every time someone has a legitimate complaint about something that ought to Just Work.
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
Parts of the HLR might be unavailable or under a thundering herd as all phones are trying to register at once. The HLR would also be involved if using EAP-SIM/EAP-AKA so that would explain failures on the Wi-Fi access points.
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
https://www.wsj.com/politics/national-security/china-cyberat...
Keep in mind who the President is.
If so, successful test.
These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks than to burn it by causing a massive, visible problem.
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
[0] https://mtlynch.io/first-impressions-of-meshcore/#testing-th...
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
I assume state on state cyber attacks are commonplace but get minimized to avoid public fear.. perhaps this will be the first notable one.
Made worse by the fact Estonia is a more networked society than, for example, the US.
https://downdetector.com/status/t-mobile/ ~ 1,600
https://downdetector.com/status/att/ ~ 1,500
https://downdetector.com/status/verizon/ ~ peaked at ~169k, dropped to 67k
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
I noticed that a little while ago too. Very bad UI. Tempted to post in r/dataisugly.
Looking at the replies some of the people who are more worried about the outage seem to be the OF models
Not judging or trying to make a point. Just find it interesting the ways things are interconnected
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
BART Defends Cutting Off Cell Service In Subway
https://www.npr.org/2011/08/16/139664637/bart-defends-cuttin...
And will happen again.