Mike Howells's Blog

Author: Mike Howells

  • Why AI Might Not Get the Power It Needs

    Why AI Might Not Get the Power It Needs

    Part 3 of a series on what I learned by accident when I started paying attention to my electricity bill.

    In Post 2, I wrote about the grid as a precision-balanced machine and ended with the Virginia 2024 incident, in which sixty data centers disconnected themselves from the grid in milliseconds because of a routine equipment fault, dumping 1,500 megawatts of load that the system had to scramble to absorb. I said the event was a preview of something larger that’s already underway.

    This post is about that something larger.

    The thing I want to convey here is hard to convey because the scale is genuinely difficult to grasp. The AI buildout that’s happening right now, in 2026, is not an evolution of the previous data center industry. It’s a different category of thing, growing at a different rate, with different operational characteristics, and the existing grid was not built for it. Neither were the rules governing it.

    I am going to try to explain how big this is, why the grid can’t keep up with it, what the companies building it are doing in response, and why none of the official solutions are likely to work in time.

    The size of the thing

    The previous generation of data centers, the ones that hosted your email and your cloud storage and your video streaming, drew power in the tens of megawatts. A typical large data center campus, in 2018, might have pulled 40 to 80 megawatts. That was a substantial customer for a regional utility but not a transformative one.

    AI training data centers operate at a different order of magnitude. A single hyperscale AI campus today is being designed for one to two gigawatts of power draw. Plural gigawatts. That is, one to two thousand megawatts per campus. Per campus.

    To put a single gigawatt in context: it is roughly the power output of a large nuclear reactor. It is also roughly the average power consumption of a city of half a million people. When a hyperscaler builds a one-gigawatt AI training facility, they are building, from the grid’s perspective, the equivalent of dropping a small city onto the system, in one location, all at once.

    And they are not building one campus. Microsoft, Google, Amazon, Meta, and Oracle are each building multiple multi-gigawatt campuses, simultaneously, across multiple states. Then there are the OpenAI and xAI and CoreWeave and other dedicated AI infrastructure companies doing the same. And the regional utilities, all of them, are receiving interconnection requests for these campuses faster than they can process them.

    Projections of how much electricity AI will consume by 2030 vary, but the credible range is roughly 300 to 500 terawatt-hours of new annual demand in the United States alone. The lower end of that range is bigger than the entire electricity consumption of California. The higher end is bigger than the entire electricity consumption of Germany. Either way, what we are talking about is adding the equivalent of one of the largest power-consuming economies in the world to the existing U.S. grid, in five years.

    This was not in any utility’s planning model as recently as 2022.

    The queue that never empties

    When a new generation project or a new large customer wants to connect to the grid, they don’t just plug in. They enter what’s called an interconnection queue. The utility studies whether the existing transmission system can handle the new connection, identifies necessary upgrades, calculates who pays for what, and eventually issues the agreements that allow the connection to proceed.

    This process has always taken time. But the data published by Lawrence Berkeley National Laboratory, which tracks the U.S. interconnection queue, shows that it has stopped functioning at the scale it now needs to operate. The current queue contains over two thousand three hundred gigawatts of proposed projects. That is roughly double the entire installed generation capacity of the United States. The queue contains, in proposed form, a second United States grid.

    But the queue does not turn proposals into operating projects. Of all the projects that entered the queue between 2000 and 2019, only about thirteen percent had reached commercial operation by the end of 2024. The rest were withdrawn, cancelled, or still waiting after years of review.

    Even projects that do get built spend an average of about five years in the queue before they begin operating. That number has been getting worse, not better, over the past fifteen years.

    For a hyperscaler trying to build a data center that needs to be operational in 18 to 24 months, this is a structural impossibility. The grid expansion that would be required to serve them, on the timeline the regulated utility process can deliver, cannot happen in time. By the time the utility finishes the studies, files the rate cases, builds the new transmission line, and energizes the new substation, the data center has already been operating, or has already been abandoned.

    So the hyperscalers stopped waiting.

    Bring your own power

    Sometime between late 2024 and early 2026, the hyperscaler industry made a strategic pivot that did not get much press coverage but is going to reshape American electricity infrastructure. They decided to stop relying on the public grid to power their AI data centers.

    This is referred to in the industry as “behind-the-meter generation” or “bring your own power.” What it means in practice is that the hyperscaler builds its own dedicated power generation, on or adjacent to the data center site, and uses that generation as the primary power source. The grid connection becomes a backup, not the primary supply.

    The scale of this pivot is remarkable. As of late 2025, industry trackers were following approximately forty gigawatts of announced behind-the-meter generation tied to specific data center projects. To repeat: that is forty thousand megawatts of dedicated power plants, mostly natural gas, being built outside the regulated utility planning process, on private timelines, by companies that are not in the power business and have never operated power infrastructure at scale.

    The poster child for this strategy is xAI’s Colossus facility in Memphis, which Elon Musk built and brought online faster than any data center of comparable size in American history by deploying dozens of portable natural gas turbines on site, in many cases before the permits to operate them had been granted. The local air quality district and the Tennessee state environmental regulators have been playing catch-up ever since. Fines have been issued. Operations have continued.

    The point is not that Musk is uniquely cavalier, although he might be. The point is that the economic logic of the AI buildout makes regulatory delays unacceptable to the companies building it. A hyperscaler that waits two years for permits while a competitor builds in six months loses the AI race. The decision tree is not really a tree at all. It is a single branch: build now, deal with consequences later. Pay the fines. Settle the lawsuits. Keep the data center running.

    If you are a regulated utility trying to integrate this customer into your service territory, you are not actually negotiating with a customer. You are watching a customer build their own utility, on their own timeline, and asking you to please connect a backup line when you can.

    The rules that don’t exist yet

    The Virginia 2024 incident I wrote about in Post 2, in which sixty data centers disconnected in milliseconds, was not an isolated event. The North American Electric Reliability Corporation, NERC, has identified multiple similar incidents in both the Eastern Interconnection and the Texas grid since 2022. In each case, large computational loads have unexpectedly disconnected, in coordinated waves, in response to disturbances that the grid would normally have absorbed without incident.

    In September 2025, NERC issued a Level 2 alert about this. A Level 2 alert is a regulatory step below an emergency, asking the industry to develop better practices for handling large computational loads. The response from the industry was, by NERC’s own assessment, inadequate. Most of the entities NERC asked to develop better practices did not develop better practices.

    So on May 4, 2026, just two weeks ago, NERC escalated to a Level 3 alert. A Level 3 is the highest level of alert NERC issues. It identifies “essential actions” that grid operators and transmission planners must take to address an immediate reliability risk. NERC has not issued a Level 3 alert about large loads before. The previous Level 3 alert, issued in 2024, was about a different concern entirely, the behavior of solar and wind resources during grid disturbances.

    The current alert reflects a regulatory body that has run out of patience with the pace of industry response. NERC’s modeling, published earlier this year, indicates that the coordinated disconnection of two thousand megawatts of data center load, which is well within the operational range of a single hyperscale campus, could destabilize twenty percent of the Eastern Interconnection. That would be a blackout on the same scale as 2003.

    But here is the part that I find genuinely difficult to absorb. The Level 3 alert does not actually require data centers to do anything differently. It requires transmission planners and grid operators to study the problem, model the risks, and report back. The deadline for the initial response is August 3, 2026. The data center operators themselves are not directly bound by the alert at all.

    The reason for this regulatory gap is structural. NERC’s mandatory reliability standards apply to entities classified as part of the Bulk Electric System, which historically has meant generators and transmission operators. Data centers, even gigawatt-class ones, are classified as customers. They are bound by their interconnection agreements with their local utility, but they are not subject to the same mandatory federal reliability standards that govern, say, a nuclear power plant or a regional transmission organization.

    NERC is working on changing this. There is a process underway to create a new classification called a Computational Load Entity, which would subject large data centers to mandatory reliability standards. The estimated timeline for that process, according to industry analysts working on it, is somewhere between three and five years.

    In the meantime, data centers continue to interconnect at the pace of new construction, under existing rules, with the kind of internal protection systems that produced the Virginia 2024 incident, on an honor system that depends on the data center operator picking up the phone when the grid operator calls.

    The honor system problem

    I want to spend a moment on this, because when I learned how the current coordination between data centers and grid operators actually works, I was genuinely surprised.

    When a major grid disturbance happens and data centers trip themselves offline to backup power, the process of bringing them back online has to be staggered. If sixty data centers all reconnected to the grid simultaneously, you would create the opposite problem from the disconnection, an instantaneous fifteen-hundred-megawatt load step that the grid would have to absorb in the other direction.

    The way this is currently managed is largely by phone call. The grid operator’s control room calls the data center operations centers. They coordinate a staggered reconnection. The data centers cooperate.

    There is no technical mechanism that prevents a data center from reconnecting whenever it wants. The automatic transfer switches that physically reconnect the facility to the grid are owned and controlled by the data center, not the utility. The utility could, in an extreme case, manually open the breaker on its side of the connection, but this is a last-resort action, not an automated safety system.

    The entire system depends on the data center operator choosing to cooperate. Which has worked so far, because the operators involved have been hyperscalers with sophisticated operations teams who understand the consequences of uncoordinated reconnection and have working relationships with their grid operators. But it depends on goodwill, scaled, in an industry where the willingness to pay fines as a cost of operating quickly is a documented business strategy.

    A senior grid planner I read recently put it bluntly: the current arrangement works because the number of relevant data center operators is small enough to coordinate by phone. As that number grows, and as the diversity of operators grows beyond a handful of hyperscalers to include hundreds of smaller AI startups, colocation providers, and crypto miners, the assumption that everyone will cooperate becomes thinner.

    The collision

    So here is where we are.

    The AI industry is building data center capacity that will dwarf any new demand source the U.S. grid has absorbed in its entire history. The companies building it are doing so on private timelines, with private generation, often outside the regulated planning process, in many cases without waiting for the permits that would normally be required. The regulatory body responsible for grid reliability has acknowledged that current rules are inadequate but cannot create new rules faster than the buildout proceeds. The technical coordination between data centers and the grid is governed largely by phone calls and cooperative agreements that depend on goodwill.

    This is the operating condition of American electricity infrastructure right now. Not a future scenario. The current reality.

    The grid will not collapse next week. The grid is more resilient than this description might suggest, because of the decades of engineering and regulatory effort that went into making it resilient. But the margin of safety that grid planners used to be able to count on is shrinking, fast, and the people responsible for managing the system are publicly acknowledging that they cannot keep up with the pace of change.

    In the final post in this series, I want to bring all of this back to where we started. What does any of this mean for someone like me, a residential customer in Missouri running a dishwasher at ten PM? What is going to happen to my electric bill, and to yours? And what, if anything, can ordinary ratepayers actually do about it?

    That’s next week.

    Next Sunday: Why Your Electric Bill Is About to Get Weirder. What the AI buildout means for the rest of us, and what we can actually do.

  • The Grid Behind the Grid

    The Grid Behind the Grid

    Part 2 of a series on what I learned by accident when I started paying attention to my electricity bill.

    After Post 1 went up, the question I couldn’t stop turning over was this: if the 10 PM cutoff on my Ameren bill maps to actual physics on a continental-scale grid, then what does that grid actually look like? Not in a metaphorical sense. Mechanically. What is the thing on the other side of my wall outlet, and how does it stay running?

    I went looking, and the answer turned out to be more remarkable than I expected, and also more fragile than I expected. By the end of what I learned, I had stopped thinking of the grid as a robust piece of infrastructure that occasionally fails, and started thinking of it as something closer to a precision-balanced machine that succeeds every day for reasons that aren’t obvious until you understand them.

    This post is about that machine. It’s also about the time it almost broke, what we did to prevent that from happening again, and why something very similar is now happening on purpose, at scale, in ways that the people building it may not fully appreciate.

    A bicycle, going exactly ten miles per hour

    The single most useful way I’ve found to think about how the grid works is this:

    Imagine you’re riding a bicycle, and someone tells you that you must maintain exactly 10 miles per hour. Not 9.99, not 10.01. Exactly 10. Forever. The route ahead has hills you can’t see coming. When the road slopes up, you have to pedal harder. When it slopes down, you have to ease off, or maybe even brake. And if at any point you drift more than half a percent off speed, the bicycle disintegrates.

    That’s the grid.

    The “speed” in this metaphor is frequency. In North America, the grid runs at 60 cycles per second, also written as 60 Hz. Every generator on the grid spins at exactly this frequency, all the time. The frequency is the speed of the bicycle, and it has to stay locked at 60 Hz the way the bike has to stay locked at 10 mph.

    What makes this difficult is the hills, which are changes in demand. Every time someone turns on an air conditioner, opens a refrigerator, or starts a dishwasher, the grid encounters an uphill. The bicycle has to pedal harder. Every time someone turns off a load, the grid encounters a downhill, and has to ease off. And all of this is happening continuously, across millions of customers, every second.

    Grid operators tolerate about ±0.05 Hz of deviation from 60 Hz before they start getting nervous. That’s about 0.08% off speed. If the frequency drifts beyond roughly ±0.5 Hz, less than 1% off, protective equipment starts tripping generators offline to keep them from damaging themselves. That’s the bicycle disintegrating. Once a few generators trip, the remaining ones have to carry their load, which usually pushes frequency further out of bounds, which trips more generators. The cascade can run faster than anyone can intervene.

    So the bicycle has to stay at exactly 10 mph, on terrain it can’t see, with a margin of error under one percent, or it falls apart.

    Now imagine the bicycle is the size of a continent.

    Half a continent, in perfect sync

    The grid I’m part of, here in Missouri, is connected to a synchronous network called the Eastern Interconnection. It covers everything from the Rocky Mountains east to the Atlantic, plus most of eastern Canada. Roughly 39 states and 5 Canadian provinces.

    Every generator in that footprint, from a nuclear plant in Florida, to a coal plant in Ohio, to a wind turbine in Manitoba, to the natural gas plant down the road from my house, is spinning at exactly the same frequency, in exactly the same phase, every second of every day.

    That sentence took me a while to absorb when I first read it. Not approximately the same frequency. Not averaged over a minute. Right now, this exact second, hundreds of thousands of rotating machines spread across thousands of miles are all turning together in perfect synchronization. If they fell out of sync, the system would tear itself apart in seconds.

    This is achieved through what’s called synchronous coupling. When a generator is connected to the grid, the grid itself acts like a giant flywheel pulling it into line. A generator trying to speed up gets resisted by the inertia of every other generator. A generator trying to slow down gets pulled back by them. All those spinning turbines are mechanically locked together, by electromagnetism, into a single coordinated machine the size of half a continent.

    That machine is what we mean when we say “the grid.” Not the wires. The wires are just the connections between the rotating parts of the bigger machine.

    The thing that coordinates the bicycle

    Coordinating this machine in real time is the job of regional grid operators. The one that handles my electricity is called MISO, the Midcontinent Independent System Operator. MISO doesn’t own generators or transmission lines. It’s a coordinator, a market operator, and a referee.

    What MISO does, in simplified terms, is run an auction. Every five minutes, MISO collects bids from generators across its footprint, fifteen states’ worth of them, and figures out the cheapest mix of plants to run to meet projected demand for the next five minutes. The cheapest generation wins. That coal plant in Illinois, that wind farm in Iowa, that gas plant in Louisiana, all bidding into a single market that picks the lowest-cost combination.

    Underneath that five-minute market is something faster and more nervous. Every four seconds, MISO sends signals to specific generators telling them to ramp up or down a little, to keep the system frequency locked at 60 Hz. This is called Automatic Generation Control, and it’s how the bicycle stays at exactly 10 mph in real time. If frequency starts drifting up, MISO tells some generators to ease off. If it drifts down, MISO tells others to push harder.

    And underneath that, faster still, is the inertia of the machines themselves. When a sudden change hits the grid, the spinning mass of every connected generator absorbs the shock automatically, for the first few seconds, before any human or computer can react. That stored kinetic energy is what gives operators time to respond at all.

    So the grid is layered. Spinning inertia handles the millisecond response. AGC handles the four-second response. The five-minute market handles the minute-to-minute economic dispatch. Human operators handle the longer-term decisions about which plants to bring online or shut down across hours and days.

    It works, almost all of the time. But not always.

    The seven seconds it almost all came apart

    On August 14, 2003, a transmission line in northern Ohio sagged in the heat of an afternoon and touched a tree. That single point of contact tripped the line offline. On most days, that’s a routine event. The grid is designed to handle the loss of any single component without cascading failure.

    But on this particular day, the monitoring software at FirstEnergy, the Ohio utility responsible for that section of grid, had crashed. Operators didn’t know the line had failed. They didn’t take corrective action. Power that had been flowing on the tripped line had to find another path, so it flowed onto adjacent lines, overloading them, until they tripped too. Now more lines were down, and the remaining ones had to carry even more power, until they tripped, until a 3,500 megawatt power surge slammed across the regional grid in a direction operators hadn’t planned for.

    What happened next is one of the strangest and most important things I learned in all of this.

    The grid didn’t fail because it ran out of electricity. There was plenty of generation. What collapsed was something more subtle. As lines tripped offline, the ones still standing couldn’t sustain the voltage that the system needed to keep functioning. Voltage started sagging across the region. Generators saw the sagging voltage and started disconnecting themselves to avoid being damaged.

    Within seconds, the cascade of generators tripping offline created the very crisis the system was designed to avoid. The grid depended not only on real power, but on something called reactive power, an invisible supporting force that maintains voltage across the transmission system and allows electricity to actually flow. When reactive power support failed, the grid could no longer deliver electricity to customers, even though plenty of real power was still being generated.

    The cascade ran across the Northeast in about seven seconds. By the time it stopped, 265 power plants had shut down, over 500 generators had tripped offline, and 55 million people from Detroit to New York to Toronto were in the dark. It took several days to fully restore service.

    The lights went out, not because we ran out of electricity, but because we ran out of the ability to deliver it.

    What we did about it, and what we missed

    The 2003 blackout fundamentally changed how the North American grid is regulated. Before 2003, the reliability standards that grid operators followed were voluntary. After 2003, the U.S. Energy Policy Act of 2005 made those standards mandatory and enforceable, with serious financial penalties for violations. The body that writes those standards, NERC, the North American Electric Reliability Corporation, gained real teeth.

    The grid genuinely got more robust. Better monitoring. Better automated controls. Sensors called synchrophasors that measure grid state thirty times per second across thousands of locations. Operators today have visibility that operators in 2003 could not have imagined.

    We have not had another 2003-scale cascade in the Eastern Interconnection in the more than twenty years since. That’s not nothing. That’s a real achievement.

    But the regulators built that achievement to defend against a particular kind of failure. The 2003 cascade started with a tree branch and a software bug. The whole regulatory regime is oriented toward preventing transmission failures from cascading into voltage collapses.

    It is not oriented toward what happens when a brand new category of customer voluntarily disconnects from the grid in milliseconds, in coordinated waves, by design.

    The thing that already happened, that no one told you about

    On July 10, 2024, in northern Virginia, a piece of relatively routine grid equipment failed. A surge arrester, a device that protects transmission lines from voltage spikes, malfunctioned on a 230 kV line. The transmission line itself didn’t go down catastrophically. The system handled the equipment failure the way it was designed to: it briefly dipped the voltage on that line a few times in rapid succession, called a multi-shot reclose, while the line attempted to recover.

    This is the kind of event that happens dozens of times a year on the U.S. grid. Operators barely notice. The grid absorbs the disturbance and moves on.

    Except this time, sixty data centers were watching.

    The data centers detected the brief voltage dips and, doing exactly what their internal protection systems were designed to do, instantly disconnected themselves from the grid and switched over to their on-site backup power. All of them. In milliseconds. Across twenty-five different substations.

    The total load that vanished from the grid in that moment was about 1,500 megawatts. The equivalent of three large power plants suddenly going offline, except in reverse: instead of generation disappearing, customers disappeared. The grid suddenly had 1,500 megawatts of generation with nowhere to send it. Frequency started climbing. Automated controls had to scramble to dial back generation across the region before something worse happened.

    This event was not widely reported in the general press. It was the subject of a January 2025 NERC incident review that I doubt anyone outside the industry has read. But here is what NERC concluded from it: the way large data centers behave during routine grid disturbances was not anticipated by the planning studies, and current grid stability margins may not be sufficient to handle this new class of customer at the scale they are about to be deployed.

    In May of this year, NERC issued a rare Level 3 alert, the kind of alert reserved for genuinely serious grid reliability concerns, warning that uncoordinated disconnections of large computational loads pose a stability threat to the bulk power system. NERC simulations have suggested that a coordinated disconnection of 2,000 megawatts of data center load, well within the size of a single planned hyperscale campus, could destabilize 20 percent of the Eastern Interconnection. Fifty million people could lose power.

    That is the same scale of blackout as 2003. From the same kind of mechanism: a voltage disturbance, cascading. Except this time, the trigger wouldn’t be a tree branch. It would be a hyperscaler’s protection equipment doing exactly what it was designed to do.

    Where this is going

    I started this post by saying I had stopped thinking of the grid as robust infrastructure and started thinking of it as a precision-balanced machine. I want to be careful about what I mean by that. The grid is not on the edge of collapse. It works, every day, with extraordinary reliability, because of decades of engineering and regulatory effort. The bicycle stays at 10 mph.

    But the conditions the bicycle has to ride through are changing faster than the bicycle is being upgraded. The grid was built and regulated for a world where customers consumed power smoothly and predictably. We are now connecting customers, very large ones, who consume power in ways that the grid was not designed to handle, and we are doing so at a pace that grid planners cannot keep up with.

    Next Sunday, I want to write about the scale of what’s coming. Not the Virginia incident, but the buildout that’s about to dwarf it. Hyperscalers are no longer building data centers in tens of megawatts. They are building campuses in gigawatts. Plural. And they are building these campuses faster than the grid that’s supposed to serve them can be expanded.

    The result is a collision that’s just starting to play out, with implications that are going to reach all the way to your electric bill. I’ll get into it next week.

    Next Sunday: Why AI Might Not Get the Power It Needs. The collision between the AI data center boom and a grid that can’t keep up.

  • What My Electric Bill Taught Me About the Coming Grid Crisis

    What My Electric Bill Taught Me About the Coming Grid Crisis

    Part 1 of a series on what I learned by accident when I started paying attention to my electricity bill.

    It started with a dishwasher.

    Earlier this year, I switched my Ameren Missouri service over to a rate plan called Ultimate Saver. The pitch was straightforward: pay less for electricity if you can shift your usage away from peak hours. I’m not particularly frugal, but I’m curious about systems, and the structure of the plan was interesting enough that I wanted to understand it.

    What I didn’t expect was that trying to figure out when to run my dishwasher would pull me into a rabbit hole about how the entire North American power grid works, and into a slowly unfolding crisis that I think most people don’t realize is happening.

    This is the first of four posts about what I found. It starts small, with my own electric bill, and gets progressively bigger from there. By the end of the series I’ll be writing about why the AI data center boom is colliding with physical reality, and what that means for everyone who pays an electric bill.

    But first, the dishwasher.

    The plan that made me think

    Ultimate Saver has two parts that work independently of each other, which took me a while to untangle.

    The first part is time-of-use energy pricing. On weekdays, electricity costs more during two on-peak windows, 6 to 8 AM and 6 to 8 PM, and less during all the off-peak hours in between and around them. Weekends are entirely off-peak. So far, so simple: don’t run the dryer during dinner on a Tuesday.

    The second part is a demand charge. This one is weirder. Once a month, Ameren looks at every single hour of my electricity use between 6 AM and 10 PM, every day of the month, weekends included, and finds the one hour where I drew the most power. Whatever that peak hour was, in kilowatts, gets multiplied by a per-kW rate and added to my bill. One bad hour can dominate the demand charge for the entire month.

    When I first read this, I thought it was a gimmick. After spending some time with it, I think it might be one of the more honest pricing structures a utility has ever offered me.

    Here’s why. The cost of providing electricity isn’t really about how much energy you use over a month. It’s about how much capacity the grid has to maintain to serve you when you need it. A house that uses 1,000 kilowatt-hours spread evenly across a month is much cheaper to serve than a house that uses the same 1,000 kilowatt-hours but spikes hard for a few hours every evening. The first house lets the utility size its infrastructure to a steady average; the second house forces the utility to build for the peak and let that capacity sit idle most of the time.

    A demand charge takes that hidden reality and makes it visible. The price signal it sends is, basically: please don’t all hit the grid at once.

    Figuring out my own house

    Once I understood the structure, I started thinking about my own appliances. The big draws in a house like mine are the air conditioner, the electric dryer, the dishwasher (especially the heated dry cycle), the oven, and to a lesser extent things like the microwave.

    The AC was the puzzle to start with, because in summer it runs almost constantly. But here’s the thing about my thermostat: it’s set to 74°F during the day and drops to 70°F at 10 PM, when we go to bed. That means the AC’s most intense work, the pulldown from 74 to 70, happens right at 10 PM, the exact moment the demand-tracking window ends. The AC’s biggest single hour of the day falls outside the window that gets billed.

    So my real demand exposure during cooling season is the AC holding 74°F somewhere in the late afternoon, plus whatever else I happen to run on top of it. The pulldown is free.

    This made everything else simpler. If the AC’s contribution to demand is roughly fixed during the day, then the question becomes: what else am I stacking on top of it, and when?

    The dryer and the dishwasher, it turns out, are completely flexible. Nobody cares whether the dishwasher runs at 8 PM or 11 PM. Nobody cares whether the dryer finishes at 7 PM or 1 AM. These are appliances we treat as “run them whenever,” but their actual draw is significant. The dishwasher’s heated dry cycle and the dryer’s heating element are both heavy loads. Running them on top of an AC that’s already working hard in the late afternoon is exactly the kind of stacking that sets a new monthly demand peak.

    So I started running them after 10 PM. The dishwasher gets loaded throughout the evening and I just start it on my way to bed. The dryer is less convenient but still workable.

    It’s not a dramatic lifestyle change. It’s a small habit shift. But it removes both appliances from demand tracking entirely, and it captures the off-peak energy rate, which is a fraction of the on-peak rate. Two benefits for one decision.

    The question that broke the dam

    After a few weeks of this, a question started bothering me.

    Why 10 PM?

    The number is so specific. Not midnight, not 9 PM, not the time the sun sets. 10 PM, every day, weekends included. It’s the same number that ends the demand window and roughly the same time the on-peak energy pricing ends on weekdays. Ameren clearly chose it for a reason. But what reason?

    The easy answer is “that’s when people go to bed.” But that’s not really an answer. Lots of people don’t go to bed at 10 PM. And anyway, why would a utility care exactly when its individual customers go to bed? Utilities don’t bill at the individual scale; they think in aggregate.

    The real answer, I started to suspect, had to do with the grid itself.

    So I went looking.

    What I found when I looked at the actual data

    Ameren Missouri is part of a regional grid operator called MISO, the Midcontinent Independent System Operator. MISO coordinates electricity across 15 states and one Canadian province, from Minnesota down to Louisiana. They publish real-time operational data on their public website: how much electricity is being generated, where it’s coming from, how it’s flowing between regions, what the forecast looks like for tomorrow.

    I pulled their load curve for a normal weekday this spring. The shape of it is striking. Demand bottoms out around 4 AM at roughly 58,000 megawatts across the entire MISO footprint. It starts climbing around 6 AM as people wake up, grows steadily through the morning and afternoon, peaks around 6 to 7 PM at about 88,000 megawatts, and then, here’s the part that mattered to me, drops fast.

    By 8 PM, load is down 3,000 megawatts from peak. By 9 PM, it’s down 6,000. By 10 PM, it’s down 10,000. By 11 PM, it’s down 14,500 megawatts from the evening peak.

    Between 6 PM and 10 PM, the grid sheds roughly the equivalent of an entire nuclear fleet’s worth of demand. By the time the demand-charge window closes at 10 PM, the grid has genuinely entered a different operating regime. Expensive natural gas peaker plants that had to fire up to meet the evening peak can throttle back. Cheaper baseload generation, nuclear, coal, wind, handles the overnight load comfortably. The system is no longer stressed.

    That’s why 10 PM. It’s not about when I go to bed. It’s about when the grid as a whole stops needing to scramble. The number maps to physics, not to convenience.

    When I saw that, really saw it, in the actual hourly data, something shifted for me. The rate plan stopped feeling like a marketing structure and started feeling like a window into a real system. My dishwasher decision was a tiny instance of a much larger problem the grid is constantly solving.

    And the closer I looked at that larger problem, the more interesting it got. And the more concerning.

    Where this is going

    I want to tell you what I found, because I think most people have no idea how much is changing right now in the systems that quietly power their lives.

    Over the next three Sundays, I’ll be posting the rest of this series.

    Next week, I’ll write about what MISO actually does. How a continental-scale power grid manages itself in real time, what the 2003 Northeast Blackout taught us about how this can fail, and why every appliance you own is connected to a machine spanning half a continent that has to stay in perfect synchronization, every second of every day.

    The week after, I’ll get into what I now think is the most important undercovered story in the country: the collision between the AI data center boom and the physical infrastructure that has to power it. The companies building these data centers are starting to give up on the public grid entirely, with consequences that will eventually show up on your electric bill, whether you’ve heard about any of this or not.

    And in the final post, I’ll bring it back to where this started. What all of this means for someone like me, an ordinary Ameren customer running a dishwasher at 10 PM, and what we can actually do about it.

    I didn’t expect to spend this much time thinking about my electricity. But the more I understand about the system on the other side of my wall outlet, the more I think it’s one of the most important things I’ve ever paid attention to.

    The dishwasher was just the beginning.

    Next Sunday: The Grid Behind the Grid. What MISO actually does all day, and why the lights have to go out exactly as often as they do (which is more often than you think).

  • Why I’m Dumping my Windows Phone

    I’m an avid Windows Mobile user going back to the Windows Mobile 6.0 days with my Motorola Q9m. I loved that phone; one of the most reliable phones I have ever used. I then found myself upgrading along the way to the HTC 8X, but reliability issues forced me to abandon it. When the Nokia Lumia Icon came to market, I jumped all over it. That was two years ago…

    The Icon came with Windows Phone 8, then eventually an upgrade to 8.1. Fortunately, the Icon made the upgrade list for Windows 10 Mobile. The upgrade to W10 mobile went flawlessly. Two bugs on the Icon were fixed for me including a speaker, which didn’t work unless I was on speakerphone and my right-channel audio didn’t work while taking videos. All fixed with the upgrade to W10 mobile. I was elated. To this date, the 20 MP front-facing camera is by far the best camera I’ve ever had. I’ve filmed all of my nephew’s cross-country and track & field events including endless dog photos. So, why would I abandon it?

    The proverbial straw that broke the camel’s back, forcing my decision to abandon Windows Phone, was my company’s decision to move to o365 (Office 365). In the way we are implementing Office 365, we are requiring a feature called Modern Authentication. With Modern Authentication, you are required to authenticate to Microsoft using something other than a password, such as an authenticator app or a mobile SMS text message. Unfortunately, Microsoft isn’t interested in developing Modern Authentication into the Windows Phone ecosystem within an acceptable time period. The Office 365 blog post (linked above) says “Coming Soon,” but that was 9 months ago. I’ve given up all hope…

    About a week ago, when my company migrated my account to o365, my native Skype for Business, native Outlook client and native Calendaring apps all ceased to function. Sure, the web version of Outlook can give me some access, but it’s not the same. I want the native client.

    So, my hand is being forced. Microsoft has made it clear that they are developing apps for iOS and Android to appease the larger market share. My particular phone on my particular platform represents a niche within a niche within a niche. iOS and Android combined for a record 99% of smartphone sales last quarter. Windows Phone has an incredibly small 0.6% market share of the smartphone market. Within that 0.6% share, Windows 10 Mobile represents only 14% of Windows Mobile usage. My Nokia Lumia Icon (929) doesn’t even crack the top 5 Windows Phone handsets! This is the new world order under Satya Nadella’s leadership. Is it a bad thing? Perhaps not. Time will tell.

    As of this writing, my upgrade date is approaching in 44 days. Sadly, my Icon retailing for $459.00 back in the day, now garners a measly $25.00 trade-in value.

    2016-08-28_14-57-48

    I will likely join the rest of the collective and purchase a newly minted iPhone 7 Plus, which should be available sometime next month. I will have all the niceties of all the apps that I need for work including apps that I’ve never been able to experience with the Windows Mobile platform.

    That being said, however, all hope is not lost. There is talk, albeit rumor, that a Microsoft Surface Phone is in development. Some say it may be released in the spring of 2017. If so, that just may be enough to pull me back in. Time will tell…

    Update: On November 14, 2016 I received my black iPhone 7 Plus. This phone is far superior to any phone I have ever owned including any Windows Phones. If Microsoft is to compete in this arena, they are going to need to hit a grand slam.

    Update: On November 24, 2020 I received my silver iPhone 12 Pro Max. 

  • Diary of a Garmin BITS Job Gone Bad

    It’s one of those e-mails that no one ever wants to receive…

    Dear AT&T High Speed Internet Service Customer,

    We want to remind you that your AT&T High Speed Internet service includes 150 gigabytes (GB) of data for each billing period..
    You have exceeded 150 GB this billing period

    What?!?

    Of course, I believe it is an error. But, when I open my daily usage chart, I can clearly see this is no error:

    Image

    So, what in the world is downloading all of this data and why did it start on Wednesday the 19th?

    I opened Microsoft’s Network Monitor and saw a multitude of requests to a Garmin subdomain caled nyc1.gdn.garminsource.net. It’s basically a CDN (Content Distribution Network) that Garmin utilizes to transfer high volume transactions such as map updates to its user base.

    I have the Garmin Map Updater service installed. So, maybe it is downloading a new map for my Garmin device. But, 30 GB/day is far too excessive even for the largest of map updates.

    I needed more tools at my disposal to determine what was happening. So, I downloaded and installed one of the best network bandwidth usage tools that I have ever come across. It’s called NetBalancer by SeriousBit. The NetBalancer desktop application allows you to view each process and how much bandwidth it is consuming. Once I opened the application, I could clearly see svchost.exe was consuming a rather large chunk of bandwidth.

    Image

    Now that the culprit was identified, how do I go about stopping it?

    I suspected that Garmin utilized the BITS service. Utilizing the BITS service is a common practice for developers to use, which saves them the time from writing their own file transfer service. BITS stands for Background Intelligent Transfer Service. It’s an easily identifiable service, which can be stopped via the Services applet as shown below:

    Image

    As soon as I stopped the BITS service, the download immediately stopped and my bandwidth consumption returned to normal.

    Another day passed and I re-opened NetBalancer and noticed that svchost.exe was consuming bandwidth again. I couldn’t believe it. The BITS service started itself up again. I even disabled the BITS service, which didn’t help. BITS would simply re-enable itself and then start itself. The activity started to feel malicious in nature.

    It was at this point, I decided to uninstall everything Garmin on my desktop. For sure uninstalling my Garmin apps would fix it right?

    Nope!

    I uninstalled everything Garmin and the Garmin BITS jobs continued to consume my bandwidth with no end in sight. This had been going on for days. So, something must have gone terribly wrong with some Garmin code somewhere.

    It was time to continue my investigation…

    I found some BITS commands available via PowerShell.

    The one I found to list the existing transfer jobs is this command get-Bitstransfer -allusers

    Image

    I then discovered there is a built-in command line utility called BITSADMIN that has all sorts of power!

    I issued this command in an attempt to cancel all BITS jobs: BITSADMIN /reset /allusers

    Image

    No luck.

    Of course, there is no reason given for the failure. But, after performing some research on canceling BITS jobs, it appears that you have to be logged in as the user who created the BITS job. So, how do you log in as NT AUTHORITY\SYSTEM? I actually blogged about this in 2011 in this blog article here: https://mikehowells.wordpress.com/2011/02/12/running-a-command-prompt-as-nt-authoritysystem/

    Basically, you open a command prompt as administrator. Then, launch the SysInternals tool psexec.exe as SYSTEM and it will launch a command prompt as NT AUTHORITY\SYSTEM. I was feeling pretty confident that this would work.

    Image

    Nope. It failed miserably. The error indicates that the request failed because the user (i.e. SYSTEM) has not logged on to the network. This was a fatal blow because the NT AUTHORITY\SYSTEM account is not designed to gain access to the network. This is usually reserved for the NETWORK SERVICE account.

    So, I decided to fire-up my good friend ProcMon. ProcMon, or Process Monitor, is another brilliantly written tool that is part of the SysInternals Suite. After launching ProcMon, I included only the process svchost.exe. I could then clearly see the folder that svchost.exe was accessing, which was: C:\ProgramData\Garmin\Core Update Service\MAP-NA-2014-40

    It was clear to me that the Garmin uninstaller did not do a good job of cleaning-up after itself at all.

    At this point, I had two options moving forward:

    Option 1) Use the NetBalancer tool to limit the download/upload rate for svchost.exe. This was not preferable as many things use svchost.exe and it would have unintended consequences.

    Option 2) Delete the C:\ProgramData\Garmin\ folder.

    I opted for Option 2.

    This stopped the BITS job and put it into an error state. At least it wasn’t downloading.

    I now have two strikes against me from AT&T. If I go over my 150 GB threshold one more time, I will be charged $10 for each 50 GB over my limit. Why does AT&T have such a low threshold for its DSL user base? It’s basically AT&T’s way to force you into their U-Verse service. Even their U-Verse service only has a 250 GB/month limit, although I hear it’s not enforced.

    I still have two remaining Garmin jobs that are sitting in a suspended state.

    If anyone has any ideas how to delete these stale jobs I would like to hear from you!

  • Allowing FTP over HTTP with Microsoft Forefront Threat Management Gateway

    I’ve been working with Microsoft’s Forefront Threat Management Gateway (TMG) since it was released back in November 2009 and it continues to surprise me…

    What I thought would be an easy protocol addition to an existing access rule turned out not to be the case.

    I was notified of an issue whereby users were suddenly unable to access a FTP site that they had been using for years. Their method of accessing the FTP site was to enter the URL of the FTP site within the address line of Internet Explorer (IE) using this format: ftp://username:password@example.com

    I found it interesting that this was working just fine with ISA Server 2004/2006 but suddenly did not work with TMG 2010. Considering that we just switched from ISA 2004/2006 to TMG 2010 it didn’t surprise me that something broke because I figured TMG is handling it differently, which it is.

    When you access an FTP site within Internet Explorer, TMG treats it as “FTP Over HTTP” instead of just plain ‘ol vanilla FTP. Since this new protocol was not defined in any existing access rules I had to go into the Enterprise-level policy and add the “FTP Over HTTP” protocol to the access rule.

    I selected my “FTP Access” rule at the Enterprise-level policy, went to the Protocols tab, selected Add and looked for the “FTP Over HTTP” protocol as I saw earlier. As hard as I looked, I could not locate this protocol!

    2-18-2013 3-03-56 PM

    I decided to open a case with Microsoft, who confirmed with me that this is a bug with TMG. Unfortunately, since TMG has been EOL’d (End-Of-Life’d) they have no plans to fix this.

    The work-around to fix this problem is to add the access rule at the array-level. Unfortunately, this means a lot of manual work especially if you have numerous arrays to manage.

    2-18-2013 4-15-21 PM

    You can simply add the “FTP Over HTTP” access rule to your existing web access policy at the array-level. Or, more likely, you’ll want to create a separate access rule especially if you do not want everyone to have access to this protocol.

    Shown here is the “FTP over HTTP Access” rule successfully added to the array-level policy:

    2-18-2013 3-28-30 PM

    When you log access to this rule you’ll notice another cool feature. You can see the developers of TMG anticipated that the password information for FTP sites are sent in clear-text and that this information may be easily viewed via the live logging session. So they remove the password during a live logging session as shown below:

    2-18-2013 3-57-26 PM

    There is one additional workaround available to you. If you do not want to create a special access rule in TMG to allow this behavior, you can make a change in Internet Explorer. Simply open Internet Explorer, select Tools, Internet Options, Advanced and scroll down to the Browsing section. In the Browsing section you will see a setting called “Enable FTP folder view (outside of Internet Explorer).” Checking this box will allow you to access FTP sites within Windows Explorer and it will not invoke the “FTP Over HTTP” protocol.

    2-18-2013 4-03-34 PM

    If you’d like to learn more about publishing FTP in ISA Server or TMG that a look at the following article. It is the most thorough discussion on FTP in ISA and TMG that I have seen:

    http://microsoftguru.com.au/2010/08/27/troubleshooting-outbound-ftp-access-in-isa-tmg-server/

    If you are interested in adding/allowing malware inspection for FTP access rules in TMG checkout the following article (normally this is not possible):

    http://carbonwind.net/blog/post/Forefront-TMG-2010-Using-malware-inspection-and-URL-filtering-for-FTP-on-access-rules.aspx

  • IP2MAC

    In all previous instances when working with Windows NLB (Network Load Balancing) I have always used Unicast. Recently, I ran into a scenario where Unicast was not allowed in a VMWare environment, which forced the use of Multicast instead. Apparently, using VMotion works with Multicast as opposed to using Unicast. Note: For a detailed description between these NLB options see the articles posted at the end of this article.

    The three options available to Windows NLB are: Unicast, Multicast and IGMP Multicast. In short, Unicast is a configuration setting which instructs Network Load Balancing to change the MAC address of the cluster adapters to the same value for all hosts in a cluster. This is the default mode of operation. Multicast is a configuration setting which instructs Network Load Balancing to add a multicast MAC address to the cluster adapters on all hosts in a cluster. The adapters’ existing MAC addresses are not changed. IGMP Multicast is essentially the same as plain Multicast except that the IGMP (Internet Group Membership Protocol) helps eliminate switch flooding apparent with both Unicast and Multicast.

    The one additional “issue” with Multicast is that you will most likely need to add a static ARP entry to your distribution switches or routers to map the NLB cluster MAC to each of the NLB cluster IP addresses.

    So, how do you find out the MAC address of your network load balanced IP address?

    If you are using Unicast, you can do this by issuing the IPCONFIG /ALL command from a command prompt:

    Unicast MAC
    Unicast MAC

    But, what if you are using Multicast or IGMP Multicast? How do you find out what the cluster MAC address is in either of these cases? Hint: IPCONFIG will not help you.

    The answer is your friend IP2MAC…

    IP2MAC is a option available from the NLB.exe command found in the %SystemRoot%\System32 folder:

    IP2MAC
    IP2MAC

    How do you execute this command?

    Open a command prompt and change your folder path to %SystemRoot%\System32 and then type: NLB.exe ip2mac <cluster IP>

    So, for example, if your NLB cluster IP address is 10.0.0.254 you would type: NLB.exe ip2mac 10.0.0.254

    The results will appear as follows:

    IP2MAC results
    IP2MAC results

    You are now presented with the Unicast, Multicast and IGMP Multicast MAC addresses. You can now give this information to your network administrator so that the static ARP entry can be made to the distribution switches or routers (assuming Multicast or IGMP Multicast).

    Interestingly, the Unicast and Multicast MAC addresses are very similar except the high-order octet for Unicast begins with an 02 whereas Multicast begins with an 03. IGMP Multicast is way out there in left field with hardly anything similar.

    Using NLB with ISA Server, Part 1: How Network Load Balancing Works

    Selecting the Unicast or Multicast Method of Distributing Incoming Requests

  • My Worst Flight

    It was Thanksgiving in 1991 and my family and I agreed that for our family reunion in Florida that I would fly everyone down in a small single-engine four-seater Piper Turbo Arrow.

    Piper Turbo Arrow (PA28RT201T)

    I had been flying for several years and I had around 200 hours of flight time and I had just obtained my instrument rating.  My instrument rating gave me the ability to fly in inclement weather and it gave me just enough confidence to fly my family from St. Louis, Missouri all the way down to the southern tip of Florida. After making a phone call to FSS (Flight Service) to check on the weather I was assured that the weather was beautiful en route and that I shouldn’t have any problems.

    The flight plan originated from The Spirit of Saint Louis airport (SUS) in Chesterfield, MO and included three fuel stops with each leg roughly two hours in length:

    Stop #1) Huntsville, Alabama (HSV)
    Stop #2) Tallahassee, Florida (TLH)
    Stop #3) Regional Southwest airport (RSW) in Ft. Myers, Florida
    Stop #4) Marco Island airport (MKY) in Marco Island, Florida.

    My brother and his girlfriend were flying in from Los Angeles, which is why we made a short stop-over in Ft. Myers airport to pick them up. This was going to be a red-eye flight so that we could get to Marco Island early and meet-up with everyone in a timely fashion.

    Below is a map of our route of flight:

    SUS-HSV-TLH-RSW-KMKY

    We departed the Spirit of Saint Louis airport just after 10:00 pm on Thursday, November 28, 1991. Shortly after takeoff, we entered IMC (Instrument Meteorological Conditions) at my assigned altitude of 7,000 ft. When we were overflying Paducah the weather conditions cleared. It was an awesome view seeing the full moon through the ragged cloud deck. The remainder of the flight was uneventful and we landed 10 hours later at Regional Southwest airport (RSW) in Ft. Myers, Florida just after 8:00 am the next morning. The most beautiful part of this trip was the flight over the Gulf of Mexico. A sight I will never forget was the sunrise over the Gulf from 7,000 ft on a clear morning with air as smooth as glass. I regret not having a picture of the view.

    After spending a few days with our relatives and enjoying what southern Florida had to offer it was time to return home and get everyone back in time for work on Monday. Our return trip from Marco Island, Florida to the Spirit of St. Louis airport was almost identical to our route coming down.

    The flight back included three stops:

    Stop #1) Tallahassee, Florida (TLH)
    Stop #2) Huntsville, Alabama (HSV)
    Stop #3) Chesterfield, MO (SUS)

    Below is a map of the planned flight:

    KMKY-TLH-HSV-SUS

    Before departing from Florida I had been making daily phone calls to FSS to keep an eye on the weather. This flight was over 1,000 miles in length so we were crossing a lot of real estate so my attention needed to be on the weather. On Sunday, December 1, I called FSS and they mentioned that a line of weather was making its way down and that I should plan accordingly. FSS would never make any recommendations but they would merely give you the information you needed and then allow you to make your own personal decision on what to do. I could see on radar that precipitation was developing to the northwest of Huntsville but it was several hours away. The weather in St. Louis also looked marginal but more importantly there were no reports of icing.

    At this point, I informed my family early that Sunday that we should leave as soon as possible to try to beat the weather.

    The one funny thing about departing an airport in southern Florida is that they have to clear the runway of any alligators before you takeoff.

    We lifted off from Marco Island at around sunset on Sunday, December 1.

    The flight from Marco Island to Tallahassee was uneventful. After refueling I walked over to the Tallahassee FSS, which happened to be located at the airport. I spoke to the FSS person on duty and he showed me the live weather radar. I could see that the weather was approaching Huntsville faster than anticipated. In fact, it was a two hour flight from Tallahassee to Huntsville and the weather was two hours away from Huntsville. I thought to myself that if I put the throttle to the firewall I could beat the weather. I had a real case of “get-there-itis.” In retrospect, if I were more experienced at the time, I would have recognized this human factor in aeronautical decision making but the pressure to get my family home override any apprehension that I was feeling at the time.

    So, I took off from Tallahassee and set the Piper Turbo Arrow’s engine power to the maximum 75% allowable instead of the planned 65% power setting. I also had a tailwind so my ground speed was a mind boggling 174 knots (200 mph). At this speed, I am pretty confident I can land before the thunderstorms start rolling into Huntsville. However, during the flight at cruise altitude, I start seeing continuous lightning in front of me on the horizon about 100 miles away. This was not a good sign. The thunderstorms were moving towards the airport faster than predicted, which meant that they were possibly intensifying. Since the airplane was on autopilot I had a lot of time to ponder what I was going to do. I reached into my flight bag and grabbed my sectional and started to look for another airport in case we had to divert.

    No more than 5 minutes went by when I looked up from the map to make sure we were still on course when, to my shock, I saw that the VAC annunciator warning light was illuminated on the instrument panel. The vacuum system had failed! At first, I went into this strange state of denial where I thought perhaps that the VAC light wasn’t really on and that it was some sort of optical illusion and that this could not possibly be happening with my family onboard. Maybe the light came on by mistake? Then, my worst fears were confirmed. All of my vacuum instruments including my artificial horizon and heading indicator started to rotate wildly. Now I’m in deep shit…

    So, here I find myself in solid instrument conditions in the clouds at night with no instruments to safely navigate except for my lonely magnetic compass and my electronic turn coordinator. Immediately, I called center (air traffic control) and advised them that my vacuum pump had just failed and that I lost all of my main instruments. The controller acknowledged this and asked me to keep him advised. At this point, I did not immediately tell my family what had happened even though they later told me they noticed the warning light was on for a long time and they were wondering what was going on. The last thing I needed was a panicking family distracting me from doing my job (aviate, navigate, communicate).

    I briefly considered continuing the flight into Huntsville on partial instruments. But watching the continuous lightning on the horizon made me think twice about that. I knew it was definitely not a good idea to land in instrument conditions with thunderstorms in the vicinity especially in an airplane with partial instruments. I knew I was going to have to do a no-gyro approach and I wanted an airport with good weather conditions.

    Because my vacuum pump failed my autopilot was inoperative. I enlisted the help of my sister who was sitting to my right to take the controls in an attempt to keep the wings level for me. But, since we were in instrument conditions this was an extremely dangerous idea. I called the controller and notified him that I no longer wanted to continue to my planned destination to Huntsville, Alabama. I now delegated the task of finding a suitable destination airport to the air traffic controller. The controller then started reading off weather conditions at airports around the region and they all had low cloud ceilings (not good with a partial instrument panel). At this point in the flight I had assumed control of the airplane keeping my focus on keeping the wings level.

    I knew I would have to shoot a partial panel ILS (Instrument Landing System) approach which is something I had never done before in actual instrument conditions. All of my training would now be put to use. The controller read the weather conditions at Montgomery, Alabama and it seemed to be the best weather around. At this point I was located around the Columbus, Georgia area so I took up westerly heading to Montgomery (MGM) about 60 miles away. My original flight plan has now been scrapped and I need to get this plane on the ground as safely as possible.

    My new route of flight now looks roughly like this:

    TLH-roughly near CSG-MGM

    Fortunately, there was no turbulence to bounce around my magnetic compass and the Montgomery airport is radar equipped and manned by an air traffic control tower. I was then given no-gyro vectors to the ILS approach at Montgomery. Below is the approach plate for the ILS 28 approach into MGM:

    ILS 28@MGM

    The flight to Montgomery went well until I was given no-gyro vectors to intercept the inbound course (i.e. “start turn, stop turn, stop!”). Since no-gyro vectors can be extremely inaccurate I was vectored to intercept the outer marker at the outer marker and the controller asked me if that was ok. Controllers are normally supposed to vector you at least 2 miles outside the outer marker. I agreed that it was ok because I didn’t want to spend any more time than necessary up in the air. Since I did not have any primary navigational instruments I had to fly solely by the ILS needle in the cockpit. I chased the needle quite a bit but I broke out of the clouds at about 1,000 ft AGL (Above Ground Level). The runway approach lights were so bright I thought I was going to go blind. The landing was uneventful and we were very fortunate in the fact that the airport had a Piper authorized service center.

    The plane was repaired the next day but the weather in St. Louis had deteriorated with freezing rain and low cloud ceilings. We were stranded in Montgomery, Alabama for two days until the weather cleared in St. Louis.

    We finally departed on Tuesday, December 3 with one stop-over in Padacuh, Kentucky and finally home to Spirit:

    MGM-PAH-SUS

    I still wonder to this day what would have happened if my vacuum pump did not fail.

    Even though the title of this blog is my worst flight in some ways it was one of my best flights. All of the hard work I put into training for this emergency paid off.

    For any student pilots or even experienced pilots you should always keep “get-there-itis” in check. Since becoming a flight instructor I use the following definition and guidelines to help recognize and avoid this syndrome:

    Get-there-itis is a tendency that clouds the vision and impairs judgment by causing a fixation on the original goal or destination combined with a total disregard for any alternative course of action.

    1. Duck-under-syndrome, not get-there-itis, is a tendency to “sneak a peek” by descending below minimums during an approach.
    2. Get-there-itis occurs when the pilot has an extremely strong motivation to arrive at the planned destination.
      1. When the motivation to “get there” is strong enough, it overshadows all perceived obstacles to completing the flight as planned.
        1. The pilot does not want to hear about things that would be grounds for delaying or canceling the flight.
        2. The pilot has the ability to recognize a potentially dangerous situation, but chooses not to since the perceived rewards of completing the flight are so great.
        3. This phenomenon is common in general aviation when a pilot is returning to his home base after a weekend cross-country.
          1. The pilot absolutely “must” be back home in time for an important commitment, and en route weather that might have caused him/her to cancel the flight earlier may be ignored on the return leg.
        4. This phenomenon is also common in military pilots who have been separated from their families while on an extended operational deployment.
          1. Aircraft that would have been rejected for any mission during the deployment because of mechanical problems suddenly become perfectly acceptable, since they are the pilot’s only means of early transportation home.
    3. As an instructor, one way for you to help your students avoid get-there-itis is to train them to consider that the flight might not be completed, and plan accordingly in advance. For instance, tell them to schedule an extra day off from work when returning from a cross-country flight, even though they probably will not need it.

    Edit: The video which most closely resembles what could have happened to me is this accident case study from the Air Safety Institute: Accident Case Study: Single Point Failure

  • Adding a Child Domain Using Windows Server 2003 vs Windows Server 2008 R2

    If you’ve ever had to add a new domain tree to an existing domain in Active Directory using Windows Server 2003 you may have already realized that you must have DNS configured properly before creating the new child domain. Put another way, if you didn’t know what you were doing you could get into trouble very quickly. With Windows Server 2008 R2 this process is dramatically simplified and the steps for DNS delegation are done for you automatically.

    Our example forest is simple with bigfirm.biz representing the forest root domain and ecoast.bigfirm.biz representing the child domain.

    The domain controller in bigfirm.biz is bigdog.bigfirm.biz at 192.168.2.130.

    The domain controller in ecoast.bigfirm.biz is srv1.ecoast.bigfirm.biz at 192.168.2.131.

    If you’ve read any of Mark Minasi’s books you’ll notice that this is the naming convention he uses.

    In the below screenshot you can see that I already ran DCPROMO on bigdog.bigfirm.biz and DNS is already configured with the DNS forward lookup zones already populated.

    Note: I had the DCPROMO process automatically install and create DNS for me for this process.

    DNS on bigdog.bigfirm.biz (Windows 2003)

    Now we’re at the point where we want to add the child domain of ecoast.bigfirm.biz to the existing forest root domain of bigfirm.biz.

    With Window Server 2003 you must create the DNS domain on the parent before you run DCPROMO on the child domain controller.

    Therefore, right-click the bigfirm.biz DNS zone and select the option to create a new domain and then enter the domain name of ecoast. You don’t have to enter any records in ecoast.

    DNS on bigdog.bigfirm.biz (Windows 2003)

    The next step is to prepare the child domain controller in the child domain.

    On srv1.ecoast.bigfirm.biz you need to point its primary DNS server to the parent DNS domain controller (bigdog.bigfirm.biz) at 192.168.2.130. If you screw-up here and point DNS to itself the child domain controller will have no way to get home to the “mothership” and report an error once you try to run DCPROMO.

    TCP/IP settings on srv1.ecoast.bigfirm.biz (Windows 2003)

    There is another minor but very important procedure that you must also do on the child domain controller (srv1).

    You must populate the DNS suffix box with the new domain that you are creating (ecoast.bigfirm.biz). If you don’t do this step then the child domain controller will not populate the DNS records properly at the parent DNS zone.

    DNS suffix settings on srv1.ecoast.bigfirm.biz (Windows 2003)

    Once all of these procedures have been done you can now run DCPROMO on the child domain controller srv1.ecoast.bigfirm.biz.

    Note: Don’t forget to allow dynamic updates on the parent DNS server (bigdog.bigfirm.biz) or else the process will fail. The DCPROMO process should warn you of this.

    What I see happen a lot with Windows Server 2003 is that it takes WAY too long for these DNS records to populate at the parent. In fact, it may take upwards of 10-15 minutes or so. Don’t be surprised if you see errors in the system event log on srv1 such as this (see screenshot below). This type of problem usually auto-corrects itself but if it doesn’t you can try opening a command prompt and typing ipconfig/registerdns on srv1 to see if it can help speed up the process.

    Event viewer on srv1.ecoast.bigfirm.biz (Windows 2003)

    After waiting the aforementioned 10-15 minutes for replication to occur and\or after manually issuing the ipconfig/registerdns command on srv1 the DNS zone on bigdog.bigfirm.biz should now look like this:

    DNS on bigdog.bigfirm.biz (Windows 2003)

    You’ll notice that DNS is not being hosted on srv1 but is instead being hosted on the parent domain controller bigdog. What if you want to have srv1 host the DNS zone ecoast.bigfirm.biz instead? You can easily do this by a process called DNS delegation. DNS delegation can be a good idea especially if you want to reduce network traffic, provide redundancy and simplify your DNS environment. There is a great KB article on how to create a child domain in Active Directory and delegate the DNS namespace to the child domain. The KB article for this is listed at the end of this article.

    From my perspective, the above procedure seems time consuming and laborious. Wouldn’t it be nice if Microsoft improved on this procedure? With Windows Server 2008 R2 your wish has come true. I get the impression that the directory services team at Microsoft took some heat for this procedure on Windows 2003.

    For the below example, everything remains the same except we are now using Windows Server 2008 R2 as our operating system.

    After running DCPROMO on bigdog in our forest root domain bigfirm.biz our DNS zone looks like this:

    DNS on bigdog.bigfirm.biz (Windows 2008 R2)

    Now, here is where things get super cool. Remember all of the steps that we went through to prepare our DNS environment before we could even introduce a new child domain into the mix?

    Well, prepare to be amazed.

    As before with our Windows 2003 example, on srv1 make sure that you point the primary DNS server to the parent DNS server (bigdog.bigfirm.biz).

    TCP/IP settings on srv1.ecoast.bigfirm.biz (Windows 2008 R2)

    Once you do that all you have to do now is run DCPROMO on srv1!

    One thing I like about the new DCPROMO with Windows Server 2008 R2 is that it automatically checks and detects that there is no DNS server authoritative for the ecoast.bigfirm.biz domain. Therefore, because it could not find an existing DNS server authoritative for ecoast.bigfirm.biz it will automatically create a DNS delegation for you. Brilliant!

    Below you can see in the DCPROMO summary screen that it will automatically create the DNS delegation for you since you did not pre-create the ecoast.bigfirm.biz domain on the parent server.

    Below is a screenshot of what the bigfirm.biz DNS zone looks like on bigdog.bigfirm.biz after the DCPROMO process completes on srv1.

    Notice that ecoast is greyed-out indicating that the zone is now delegated.

    Delegated DNS on bigdog.bigfirm.biz (Windows 2008 R2)

    After logging into srv1, DNS was installed automatically and the ecoast.bigfirm.biz DNS zone was created and populated with all of the DNS records. No errors in the event log and everything just works and works immediately.

    DNS on srv1.ecoast.bigfirm.biz (Windows 2008 R2)

    They say the devil’s in the details and Window Server 2008 R2 does not disappoint. Below you can see that the DCPROMO process automatically adjust the primary DNS server on srv1 to itself and points its secondary DNS server to its parent DNS server.

    TCP/IP settings on srv1.ecoast.bigfirm.biz (Windows 2008 R2)

    One final note I should mention is that it is no longer required to populate the DNS suffix on the child domain controller srv1 as we were required to do with Windows Server 2003.

    How To Create a Child Domain in Active Directory and Delegate the DNS Namespace to the Child Domain
    http://support.microsoft.com/kb/255248

  • Reading a Remote Registry Key Through Scripting

    I’ve been working a lot lately with SCCM DCM (System Center Configuration Manager Desired Configuration Manager).

    If you’ve worked with ConfigMgr you know how powerful the tool is. The DCM portion of ConfigMgr is particularly powerful when scanning collections for compliance against a set of baselines (comprised of configuration items).

    The one thing that you quickly realize with either ConfigMgr or DCM is that you need to script a lot of stuff to get what you want. DCM will allow you to use three different scripting frameworks: PowerShell, JScript, or VBScript. For my situation, PowerShell is not an option because the target servers must have PowerShell installed, which is not a guarantee. So, I chose VBScript.

    One of the scans we are performing is to check for the existence of a registry key and key value. The registry entry is the following:

    HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\IniFileMapping\Autorun.inf
    Value: (Default)=@SYS:DoesNotExist

    The above registry key and value is one of those Windows Secrets that prevents AutoRun attacks.

    Note: Details of the AutoRun attack and how to prevent it is listed at the end of this article.

    Reading the local registry via scripting is relatively straightforward. Using the WshShell object’s RegRead() method you can display the value located in the above registry hive by running the following VBScript.

    Set ObjWshObject = WScript.CreateObject(“WScript.Shell”)
    strResults = objWshObject.RegRead(“HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\IniFileMapping\Autorun.inf”)
    WScript.echo strResults

    If you’re wondering how to run the above script you can take the above green text and paste it into Notepad (or my favorite Notepad++) and save as RegRead.vbs and then execute it by double-clicking the vbs file.

    Note: If this registry hive does not exist when you run the script you will receive an error.

    The question now becomes how do you run this script against a remote system to see if this registry value exists? It’s too bad we can’t just wave our magic remote wand and make VBScript magically do this. If only it were that easy…

    To get VBScript to work remotely you have to invoke WMI (Windows Management Interface). WMI is a massive topic and way beyond the scope of this article. Suffice it to say it is the magic you will invoke to gain access to remote stuff.

    The first problem in trying to execute the above script against a remote system is that we are interrogating the registry for subkeys when we actually want the default value of the key.  The second problem is that WMI’s StdRegProv (WMI interface for remote registry access) is really hard to use.  It is full of all kinds of pitfalls because the results depend upon the type of data found in the registry.  For example, if the default value is not set it only returns a scalar value (single value) as opposed to returning an array (multiple values). Also, you need to determine the value’s data type before you can read that value.  That is to say every value type requires a different method to extract its value. Wow that just turned difficult fast. Microsoft should examine this portion of StdRegProv because it is unreasonably complicated. However, my belief is that Microsoft’s focus is more on PowerShell as a solution as opposed to using VBScript so it is what it is…

    In our example, life is a bit easier because we already know that the default value is going to be a string value.

    Cobbling all of this extraneous information into a script to gain what we need looks like this:

    const HKEY_LOCAL_MACHINE = &H80000002

    ExistOrNot = “Key does not exists”

    strComputer = “.”

    Set objReg=GetObject(“winmgmts:{impersonationLevel=impersonate}!\\” & strComputer & “\root\default:StdRegProv”)

    strKeyPath= “SOFTWARE\Microsoft\Windows NT\CurrentVersion\IniFileMapping\Autorun.inf”

    objReg.GetStringValue HKEY_LOCAL_MACHINE, strKeyPath, “”, strValue
    if instr(strValue, “@SYS:DoesNotExist”) <> 0 then
    ExistOrNot = “Key exists”
    else
    ExistOrNot = “Key does not exist”
    end if
    WScript.echo ExistOrNot

    Without going into the gory details of the script the magic really happens in the last section when we issue the following function : if instr(strValue,”@SYS:DoesNotExist”) <> 0.

    The Function will return the position of the first occurence of @SYS:DoesNotExist within the variable strValue. The Function will return a value of zero if @SYS:DoesNotExist is not found. Therefore, if the value returned is not zero then our key exists as our script shows above.

    Scripting is akin to playing an instrument. Just like in anything the more you do it the better you become at it.

    One quick trick prevents AutoRun attacks
    http://windowssecrets.com/2007/11/08/02-One-quick-trick-prevents-Autorun-attacks

    Notepad++
    http://notepad-plus-plus.org/

    Microsoft Script Center
    http://technet.microsoft.com/en-us/scriptcenter/bb410849.aspx