There's a YC video where they tell how everytime they visited the data center, Facebook servers seemed to creep in and multiply.
So I guess they just bought a lot of servers
[deleted]
Sir, that’s called a stateless web server. It has nothing to do with PHP
Yeah then I'd argue that the actual scaling comes from where and how the state is managed.
My guess is they created a distributed database engine just for that (CassandraDB).
Also memcached
Depends on the architecture, it's not php doing
But what about the dispatch of queries ? The databases ? Php is only a part of the issue.
well, everything scales as a proportion to the number of servers you have so that's a trivial claim.
php just forces you into shared-nothing architecture but you can do that without php. you just don't tend to do it because it leaves a lot of performance on the table.
They're building an AI data center nearby at the moment and the the building is starting off at the size of an entire Amazon warehouse
Nobody tell them it was also written in PHP
Still is, they actually developed their own JIT to make it run faster https://en.wikipedia.org/wiki/HHVM
And if someone wonders why they didn’t just rewrite the codebase — rewrites are risky, slow, and expensive. Instead, they made PHP faster with HHVM. Pragmatic move.
Of course at the time they could have written it using Java JSP, and then there wouldn't have been any need to write their own VM. You also would have gotten static type checking, threads, and prepared statements back in the year 1999, instead of waiting for PHP to reinvent the ideas badly.
Everyone likes to shit on Java, but the verbosity is not bad, unless you choose to use a bunch of silly enterprise patterns.
IIRC PHP was at the time much easier to load balance because each request is handled by its own separate application instance, so all you needed to do was to have a way to share session data. Whereas Java solutions (again, at the time) were difficult to scale horizontally that way.
Happy to be corrected on this, but that was my sense at the time.
JEE was a fucking mess. There's a reason nothing looks like JEE today other than JEE.
The individual technologies sucked less over time but ultimately the whole model of having a huge application orchestrate everything was simultaneously too much and too little. Those insane app servers weren't nearly enough for the type of system that uses kubenetes today but were also far too much for most simple use cases.
If you just had an easy way to launch JSPs without having some crazy JEE application server behind them it would have been used more.
I used Java for over a decade before switching to Scala. Never used any J2EE, other than JSP, the Servlet API and maybe some other not terrible APIs I am forgetting...
We launched JSPs using Tomcat. It was not at all crazy. Maybe a little more involved than setting up a LAMP stack (which is also not trivial, unless you rely on it being preinstalled in a distro).
It is worth keeping in mind Tomcat was a demo technology meant to show how one small part of JEE should work. It was always covered with "do not use this in production, this is only a demo and you absolutely need all the super secret sauce extras that JEE provides" type warnings.
Tomcat became used in production a lot because a stripped down demo project was much closer to what people wanted. It is the perfect example of how bad JEE actually was.
which is also not trivial, unless you rely on it being preinstalled in a distro
That is how 99% of web hosting was delivered back then.
unless you choose to use a bunch of silly enterprise patterns.
The problem with Java is the silly enterprise patterns are a core part of its ecosystem's identity.
Instant legacy code
Modern Java (17) is not nearly as verbose and shitty. Things like Guice and Jakarta have made DI significantly better and modern frameworks like Micronaut have further improved on this.
PM: 'Sorry, money's too tight to upgrade or rewrite legacy enterprise apps. Best we can do is Java 8.'
I think this is why Java gets such a bad rep tbh. I had the misfortune of working on a legacy JDK8 code base with a bunch of ant build scripts for 3 months; complete and total nightmare.
Fortunately, I have had the opportunity to develop two services from the ground up in JDK17, one using Spring with Guice, and the other with micronaut.
The latter two services were way more fun to write AND maintain, the micronaut one especially.
A few years ago, I had the misfortune of working on a PHP app written in PHP 5.5. People like you just assume there isn't legacy crud in the world of PHP...
I also remember being in a meeting of volunteer nerds working on the website for a college radio station.
They needed to upgrade the ancient website from PHP 5, the problem is that everything was going to break.
In the Java world, I constantly upgrade the JVM with almost no problems. This is because the language was created by professionals who consider backwards compatibility to be very important.
I work for a very large company, and I've upgraded the VM for our Scala apps from 8, to 11, then 21 and soon 25.
Large orgs might be afraid to upgrade, or can't because they use some fancy framework and it would be too painful. But lets not pretend that doesn't happen with PHP...
Discussing about facebook coding decisions decadesa ago
.
Modern
bruh
More importantly two very different skill sets and focus areas allowing two different teams to work on the problems independently.
One team continues to delivery customer facing functionality while the other team focuses on core infrastructure instead of one team not delivering anything visibly new for a year or more.
TBH when they hit the slow aspects it was basically a fully fledged product.
A rewrite could have meant MySpace could have pivoted at that time and likely captured the space; especially if they were aware it was happening.
Instead they simply addressed the performance concerns, whereas in a very complex way it was less complex than burning resources on a rewrite.
Today... I seriously wonder what percentage of functionality is still on PHP+HHVM considering the tools at the disposal now they likely have their platform fairly well segmented.
They also used Ajax to query data from the backend and render it on the frontend.
Back it that day, you'd call it ajax. Long before SPAs have been a thing (Facebook also invented react but years later.
Back in the day, it was all about Caffeine and Late-Night Coding, no fancy cloud stuff
"caffeine"
Caffeine for the nose clearly
< Laughs in Harvard >
NyQuil according to his desk
He didn't, actually
Did people just forget that Facebook started as a small site and didn't immediately spawn in as a corporate megabehemoth?
I think the joke is more that some people over engineer their small site as if it were a megabehemoth from day 1.
He actually gave a lecture about how Facebook started, he gave not just the technical details but also the business side of things. Really fascinating story.
Anywhere I can watch or read this?
Not sure which lecture he’s referring to, but it might be the CS50 lecture he gave years ago, although I’m not sure as I never finished that
Highscalabilty.com has a ton of articles on Facebook. They are also a great resource for reading about those early days of the web. https://highscalability.com/tag/facebook/
I remember talking to an engineer at Hotmail back around 2001 and he was saying they had to format the hard drives only for the inner rings of the disk because it improved seek time.
If you do that correctly, it’s not any more expensive than the alternative, and it’s not any more effort than the alternative.
Why not prepare for the outside chance that it happens? Better that than to be bitten by influx-led site crashes and be forced to re-engineer your infra.
The meme is basically saying “Zuckerberg didn’t need these tools before they existed, why do you need them?” And the answer is “if they’d existed when he was building Facebook, he would have used them.”
Time to market...
I once joined a startup thinking it was the very beginning of development based on their progress. Turns out, they had spent the past two years setting up a really fancy cloud deployment process back in the early days when we didn’t have nearly as many tools as we do now. They were using JVM languages, and had an extensive suite of automated tests setup.
That company doesn’t exist anymore.
And this is why sometimes you need a product owner/manager to tell us nerds that we don't need to plan for 2 million users on day 1, we need to plan for 10000. And then you need us nerds to say okay, but we need to make sure we can somewhat reasonably rewrite it later if we ever succeed.
A good environment consists of both of these sides. Sometimes my department goes way too deep into the weeds when the product will never scale that far. And sometimes product people tell us "just do it fast, we only have 2 million people, how hard can it be".
Exactly. Or rather experimentation speed. Engineers sometimes think that business is an exact science. The truth is that until you find market fit you don't know what the heck you're doing. You're just throwing shit at the wall and hope it sticks. You need to be able to throw enough of it, fast enough, until your money runs out, to have a chance to find the thing that works.
Because it will slow you down. Losing a year of development in the early years of a startup is huge.
You’ll also find you aren’t the only startup with that idea. Someone else who gets traction before you has a greater chance of winning out.
Getting customers means getting investment which means hiring more engineers. Throwing engineers at a problem is not an automatic way of fixing scalability. But it does help. A lot. It allows you to have people work on say just the scalability of the DB, instead of flip flopping between DB / bugs / regular features.
Lol AWS is 10-1000x the price of a basic $5/mo VPS which can handle 99.9999% of hobby websites which only get 1-2 visitors per hour at most.
Well,
It kinda is... I've seen a few projects run out of budget due to VP being set intimidatingly high, mean while generating no profit to refill budget in any capacity. Let alone projects than never fully lifted off, due to not having the budget for marketing. Dev money goes fast, so if the strategy is shitty, you're out to fail.
I blame the media for creating this idea that you launch the product and go on never-ending vacation due to being a multimillionaire afterwards.
But even when it was a small site, it 'scaled out' by having separate servers per school.
Coders today wouldn't believe the things we used to do with PHP
Always called PHP my cheap slut. Not a beauty, but it would do anything.
Still my language of choice for saas, I can do anything with it. Laravel though, I’m not young enough to suffer og php anymore
I couldn't afford PHP. I had to rely on server-side includes and cross-site scripting not being recognized as a threat yet.
I still use it. I even use it instead of shell scripting on my system. It's the first language I learned (I self-taught in 2001), and by god I'm going to use it until the day I die.
I remember back at my first programming job there was this empty for loop that couldn't be removed because without it you got a segfault or memory error or something, no one had any clue why. It was just one of those weird PHP things along with the comment that if removed caused an if statement to be skipped which definitely shouldn't be possible
Part of me actually misses using PHP because I got to be so good at it that I always had a way into a project.
Coders today wouldn’t believe the things we did with VBA in Excel.
Guess he just used pure coding magic back then
2005 was when 40% of Americans were still connecting through dial up lmao.
People just had a little more patience back then.
In my company we're doing 'performance improvements' because some pages are taking 2 seconds to load. People has tiktok brain and anything not immediate is garbage.
Other side of that coin is modern websites dumping multi megabyte responses to the client just to render a simple page of text because the entire site is bloated to the gills with scripts. Because when everyone is on fiber you can get away with it.
The problem is not everyone is on fiber, even today
2sec to load IS garbage. Sub 1 sec used to be the norm, but since all these shitty node/JS frameworks 2 sec for whatever you do is the new norm.
Ok but a 2 second load time is genuinely awful lol
Also, time spent connected per user would be much lower. No smartphones, you only went on Facebook at home, at your desktop computer, which you might have to share
Websites also were a lot less dynamic and more text heavy.
A lot of memcached
Memchad
My hot take: You don't need most of the cool tech stacks and serverless BS. Most of the projects will die before you need them, and by the time when you'd actually need them, you'll have enough investor money to hire those who can do it for you.
This realization hit me real hard recently. Once your business can not be handled by a single postgres instance you can just sell your shares, live on a yacht for the rest of your life and let some some team of wizards take care of migrating your shit to scyllaDB.
To be fair, that’s a relatively recent phenomenon (I’d say at least 2010-ish onwards). Back when SSDs basically didn’t exist in the server space for cost reasons, Postgresql hit hard limits around maybe 10k disk IOPS if you were running some massive RAID array, which with all the bookkeeping it did translated to maybe 1-5k “simple” transactions per second, and that’s on a pretty meaty multi-socket server from that era. You’d want an assload of RAM to keep the entire hot set in block cache (bumps up TPS to 10k+ on huge multi-socket servers) plus read replicas for read-only transactions and failover. Sharding was still fairly common before you were at the point you could dump your shares and retire, now that’s not the case because a single Postgresql machine can reasonably handle 100k “simple” TPS with direct-attached NVME SSDs and AMD EPYC dual-socket servers, iirc it can go much higher still if your working set fits in RAM (I’ve seen 1M+ TPS on a single machine in-mem before, although that was a pretty contrived experiment).
Even amazon got tired of the shit they were pedaling and went back to Monolith for their own shit.
If you code without your IDE full screen I don't trust you
lol I feel singled out, I hate having anything full screen unless it’s a video. Browser, IDE, notepad, etc. are always not full screen for me
In windows im straight full screen but in mac having shit fullscreen just feels off
What the hell, I'm not alone in this?
My first three years as a programmer. 2006-2009, I used only vim.
My brother in nix, it's 2025 and I still only use vim
...then you figured out how to save and exit?
What if I dual window with it on one side and Google on the other?
Real programmers have 5+ monitors. /s
Based
Wait do you mean literally full screen or just maximized?
Tell me you are not a programmer with telling me you are a programmer.
Coding on windows I presume?
He summoned aliens
More servers. More servers. Less media focus. Less data collection. Tracking across other sites not as prevalent yet. Fewer platforms. More downtime expected from users.
Right? It’s just shitting out some html. Probably wasn’t even doing Ajax shit yet.
“How did he scaled”
Sticky load balancing was magic. You could cache locally instead of trying to build huge databases or regional caches.
The first version of facebook had a separate database for every college. So if you had friends at a different college you were out of luck.
People in 50 years: how did he manage to do it without vibe coding
Can you Tell me what vibe coding is?
Ask gpt
Facebook had a tiny fraction of the feature set it has now is how.
imagine if he got laid back then. the world would be a much better place.
He was with his wife before Facebook, that plotline was made up in the social network
You’d think he’d be rich enough at this point to afford chairs without cum stains on them
On a serious note, you must read Facebook blogs. If you go back to their blogs from late 2000s, you will find detailed low-level details on how they scaled Facebook.
For example, check this blog from 2008 - https://engineering.fb.com/2008/08/20/core-infra/scaling-out/
Everyone: windows is so bad for development even with wsl, winget and windows 11 dev mode.
Zuckerberg: I developed Facebook using windows XP.
Probably on WAMP
Not sure, the photo in the post is showing windows xp. Sooo
The 1.0 of XAMPP is 2003, and i guess i always called in WAMP. Ohh well, ha.
He asked to actual engineers
I guarantee all that shit only makes things harder
I attended a talk by the Facebook CTO* in the 2010s at SXSW about this. He explained that the biggest gains came from setting up caching servers (Redis) and arranging their data center racks so that web servers were on the same racks (and network switches) as the cache servers, API servers, and database servers (rather than the initial design, where they were segregated to their own separate racks).
*or someone equally as knowledgeable from the company - it's been a long time ago.
It’s like how in the cartridge era they had to fit the entire game in 2mb so they had to do tricks. Now it can just be as big as they want.
You steal it from someone that did the hard work and claim it as your own. Just like many others did. MS-Dos (where Microsoft got big with) was invented by IBM. Soooo…..
They weren’t scraping up users data and tracking them back then, so the site was much more efficient?
> Serverless architechture
> Look inside
> Servers
Lol well the original Facebook was so much simpler. Not the Advertisement behemoth it became
He didn't. That's the whole thing. "Scale" now and "scale" back then are not the same thing.
"Scale" now is effectively clueless business people demanding the system would be scalable effectively indefinitely, even if their app never reaching even a million users.
"Scale" then is a bunch of IT guys deciding how far they can stretch it before it shits the bed, to secure enough funding and rewrite the whole thing before that moment is reached.
Sometime I wish I could work at a place where we stop taking designing for scalability in advance and just address scalability when those issues actually arise. Over two decades, I feel like the amount of work I put into preemptive concerns on scalability is extreme compared to the amount of work I put into retroactively addressing them, like for example in a legacy project.
I've worked only twice on something where we knew it's important because the platform might be overloaded at launch day. And they weren't...
My old work built several racks for Facebook in the late 2000s fully stack with servers, then his company started OpenCompute to build custom server hardware it’s a pretty neat setup. I haven’t looked into their stuff since the late 2010s so I’m assuming they are still chugging along with that along with the new software side of things
he stole it, like the rest he stole. dude's a cunt.
Been in IT for 3 years with a degree. I don’t know any of these words
Monolithic code
He had many coding bitches which he Steve Wozniak’d into obscurity
You wouldn't believe it, but they actually used servers.
Devs these days can't comprehend actually knowing how to create from scratch.
Make no mistake, I would take using libraries over working from scratch any day. But it was beneficial to my understanding on a more comprehensive level.
Scale to what? Wikipedia says it had 1.5 million active users (logged at least once in 30 days) by the end of 2004. That's not even 1 login per second. I don't think that's a lot of scaling...
I code video games, but I’ve tried my hand at web apps and working with other groups and the shit we wasted the most time on was talking about what bullshit technologies we needed to implement, and a lot of these are on the meme.
How about you make something good first then worry later? But what do I know
The Job market just sucks, so they keep add more requirements of things you have to know
[deleted]
Version control existed before git :)
version_control_final_final_FINAL_v2
The one true version control ;)
Also Facebook doesn't even use git actually...
Our websites loaded slow and we were cool with it.
He build some of those solutions because it didn't scale
Using something called "brain".
It's amazing how much overhead you can eliminate by avoiding Kubernetes, Serverless Functions, serverless Redis, managed auth service, Rust, serverless edge replicated database with realtime sync, Apache Kafka, systemd and fucking storypoints o.0
I don't understand a single word in this post.
Someone learned some buzzwords
i mean, he stole it
He stole other people code and/or ideas. That's how.
IIRC didn’t Facebook help invent a bunch of that stuff so they could scale out?
check out this interesting timing between DARPA shutting down their surveillance program and facebook's launch date https://en.wikipedia.org/wiki/DARPA_LifeLog
well he successfully parsed html with regex
He didn’t even wipe the cum off his chair for this photo
He used PHP to generate dynamic html pages on the server and when they reached scaling issues they made the obvious choice to scale their servers by building their own php virtual machine with a JIT compiler.
LOL someone said it
Pretty hardcore though imo
Yeah I joke around calling 2000's programmers chads for favoring vertical scaling (scale-up) solutions, but in reality horizontal scaling (scale-out) solutions were only just entering an early adoption phase in the mid-2000's and became mainstream (for new architectures) in the 2010's.
They were just waiting for Richard Hendricks to invent the middle-out compression algorithm.
I’m more impressed by the other cofounder* knowing how long it took to jerk everyone off in a room. Dude had the math to prove it.
*Hendricks disputes the claim he was a cofounder.
This has gotta one of the best scene of recent years. Zero hesitation.
First time I saw it I couldn’t stop laughing because it was just insane.
Erlich is a dead.
Too bad his company failed. But I ve heard he is a good teacher.
Looks like he landed on his feet
"PC & Amiga" hahahaha
I think we've swung too far in the direction of horizontal scaling though. Instead of leveraging the insane performance of modern processors, we deploy everything to single core containers where that single core is shared between containers and having to run a full OS stack for each application. And then when we hit performance bottlenecks as of course we would, then the answer is to spin to a dozen more containers. Totally ignoring just how inefficient it all is, and how VM host servers are sold based on core counts rather than actual performance. They could be 1.x GHz ARM cores when we have the technology for 5.1 GHz x86 cores that will run circles around them in performance.
And then there's serverless functions where for the sake of easy horizontal scaling, we build applications where 90%+ of the CPU and memory usage is entirely in starting up and shutting down the execution environment, not our actual code.
So many applicantions architected for horizontal scaling and need horizontal scaling as a result when if they had been kept simple, vertical scaling could have handled their needs.
Tldr; We got a shiny new tool in our toolbox and its a very cool and powerful tool in the right situations, but it's the wrong tool for every situation and that's how we're using it nowadays.
Compute is cheap, engineers are expensive. We usually pick the cheapest available solution.
Problem is when team leads say "We are optimizing for engineering time," then turn around and set up Kubernetes and Kafka, and break a simple CRUD app into 15 microservices.
Depends on the situation and sadly I’m cheap (compared to the us) :(
My team’s cloud budget is 2x the payroll for the team so, maybe not always.
> having to run a full OS stack for each application.
That's simply not true. Unless you mean virtual machine instead of containers.
Wait, you think each container runs a full os stack?
Yeah it was painful to share state between multiple instances so it was always easier to beef up and scale vertically until horizontal scaling became more approachable or you rearchitected to handle it. It wasn’t easy if you didn’t start out either horizontal scaling in mind.
Yeah, I’m super nostalgic about this era of web development. I mean, FUCK EVERYTHING about it, but also… man, I miss it.
Edit: Why is nobody mentioning 1) Zuck’s nasty goon chair or 2) the Java dev sucking on his finger?
There certainly was a charm to just serving page that didn't infinitely scroll or require using the shadow DOM or virtual DOM, and we weren't pre and post processing our CSS.
.... but I think about 1/3 of my early career was making sure forms worked correctly.
tbh I still hate infinite scrolling.
I remember when every artist I knew used tumblr exclusively and scrolling through their pages would crash firefox.
You can still crash a lot of browser tabs by just scrolling down an "endless" page long enough. At least nowadays the crash is limited to a single tab.
I manually close video players on the Reddit app, because if you scroll past too many, even with them paused, it'll run out of memory and try and take my phone with it.
But that might be a me problem. I enjoy getting high, sitting in my hammock and vibing on a nice Sunday like today. Most people will never scroll far enough to break it unless they're looking at porn lol
I had to optimise a web page that presented info from a database many years ago, it worked fine for the ten or twenty rows in the test database but slowed down exponentially to where scrolling was taking 10 minutes or more to refresh on the production 2 million rows. The usual web devs said "that's just how it is with big databases" and me as the new guy measured a few things so I knew where all the slow was, then added basic paging to it so it wasn't trying to form a web page with 10000 rows and it absolutely flew.
ISTR a logic error meant that for one page it was querying for "the first row" then "the first two rows" then the first three etc. until they got to the desired length and created the page, hence the exponential slowdown.
They were pleased but didn't put me in the web dev dept, as apparently they were quite annoyed.
"How dare you make us look bad by fixing the issue. That's not how we do things here"
Exactly that. I think they were really old school DBAs who were only used to producing full reports and didn't really want anything to do with the new fangled web stuff.
Because the whole point of that web app was to provide online access to their database it had been sent their way.
iirc tumblr would still show pages if typed in like blog.tumblr.fom/page/number, and blogs also had archive view with all the posts
IE6 can choke on a bag of dicks
"Can't call undefined function on line 24" OF WHAT FILE??
Shadow DOM sounds like some kind of underground kink circle.
What makes you think it’s not?
People who like to deal with web front end certainly have a weird fetish
Perl did it ahead of time.
That was Hack, right? I wrote a ton of Hack back in 2013.
Yeah hack is meta’s spinoff of PHP
And it inspired some ideas in React that still live on
In extreme circumstances, the assailants can be stopped by removing the head or destroying the brain.
The original HipHop Virtual Machine (HHVM) ran standard php but has since diverged to mainly support Hack (Facebook's php extension).
I joined in 2007 and no joke it was not just PHP, but procedural. No static html pages. Some new hire came in one day and made photo.php use async request and site cpu usage fell in half across the tier. Those were the days.
In truth, they didn’t want to be Friendster. Performance was always a priority.
I did something similar when I started at a dotcom where the P in LAMP was Perl by just installing mod_perl it was a 95% reduction in cpu utilization. heh
After reading this comment I realized I dont belong in this sub because I understood 0% of that.
Nonsense! Anyone who is interested in programming based memes belongs here. A nice trick if you don’t understand something technical is to copy paste into ChatGPT and ask it to explain at a beginner level. I’ll also note that knowing how facebook scaled in its early years is not really relevant to 99+% of programming tasks today.
Just wanted to say I love your attitude that's all.
Hell yeah man I’m absorbing all kinds of programming knowledge from this sub and chatgpt
Building my own program in python has been a really fun and educational experience.
I still can’t produce code from scratch, but I can read the chunk ChatGPT gives me and go “hey isn’t that variable supposed to be X and not Y” and I’m starting to understand the loops and logic better.
It’s insanely addicting! I can see why people get hooked into programming
That’s awesome. I would be very interested to hear more about your learning experience, especially how you’re leveraging AI. I had a different set of tools available to me when I started learning programming, so I expect your journey to be unique.
I’d be happy to tell you more but I’m about to be busy with the fam for a few hours, but in short -
I told chatgpt I’ve never coded before but I have a few ideas for stuff to build.
It told me all the stuff to download and install (Visual Basic, Python, etc), then it started giving me code chunks and telling me how to save them all.
I have it in my project custom instruction “User has never coded before. Speak as if the user has never seen code before”, and it works well 99% of the time.
At first I wasn’t absorbing anything since it was wayyyy too overwhelming, but now after a few months, I’m starting to see how it all works.
Again, I could NOT produce code from scratch right now. But now when I see a name error or something, I can instantly identify what it’s talking about and what we forgot to add to identity something
It’s very fun!! But also mentally exhausting haha. Keeping up with every variable, double-checking chatgpt on every code chunk, etc
Dont worry, im sure that the "php virtual machine with a JIT compiler" its something that very few can do. And most that think they can, would not.
More people than you think could write a compiler if they bothered to learn. It's not terribly difficult, everything needed is taught in undergrad CS or CompE.
Writing a JIT compiler is a bunch more work to mabe performant, but there's no big conceptual leap needed.
Writing a VM is easier than making a soft-core processor of your own design but existing ISA in an FPGA, and that's an undergrad CompE task (at least it was for me).
PHP is best avoided.
So I agree, most that can, would not.
Hip Hop Move Fast and Break THings
Technically, the evolution was from PHP -> C++ transpiler (HPHPC) -> JIT VM (Hack). The latter transition wasn't for perf and actually was slower initially by some decent margin but instead because of a couple factors - principally that people kept checking in broken code (local dev was PHP because the compilation process was too expensive).