The biggest problem I've seen with architecture diagrams is they fall out of sync with the code base. In my opinion, automatic generation of these diagrams is necessary. Otherwise, teams have no way to know whether the picture in front of them accurately represents the latest state of the system.

The upshot is that diagrams have to be maintained in conjunction with the code and application architecture, just like textual documentation. There’s just no alternative to simply putting the work in, and making it a regular part of change management.

Paul alludes to it in his post about Healthcare.gov from the other day, but this was apparently one of the major parts of the failure there - endless lines of code generated from UML diagrams, making reading the code hard and things like “can we add a trace statement here” difficult.

Optimising for diagrams-as-source-of-truth has drawbacks for debugging and maintainability of the running code

https://www.pauladamsmith.com/blog/2023/10/the-10-year-anniv...

You can have what you're asking for if you agree to a predefined architecture and everyone agrees to write code that way.

Schematix generates diagrams (models) from code which can be entered via the web interface, or from a remote command line or scripts.

Diagrams are rendered on-the-fly from queries called "topological expressions" run against the model. The model must be updated as IT workers change the environment, but since diagrams are generated from code, they always reflect the most up to date information from the model.

  bool mystery_func(int i) { // define a new function
    int x = (int)(i / 2);    // x is i divided by 2 and truncated
    int y = x * 2;           // y is twice x
    return y == i;           // return true if y == i
  }                          // end of function

In my experience, auto-generated documentation in general does a poor job.

That said, such automation is improving by virtue of better AI, that understands and cross translates people's languages and computer languages.

- planning/outlining a solution

- documenting/insight into the system

I disagree that second can't be autogenerated - I've used class diagrams, database schema diagrams etc. to visualize projects, a lot of the time over the documentation - precisely because I can trust the generated diagrams to reflect current state.

Documentation is nice for context but I'd take good visualisation tools over most documentation I've seen on projects I've worked on.

Tooling to connect/validate documentation against code is non-existent, in my world at least. Maybe LLMs can change that down the line - have PR review against docs run as a part of CI/CD pipeline.

The question is: why is this the case?

IMNSHO, the reason is that we don't have a way to express architecture in or as code. Instead, we have to compile the actual architecture of the system into one that is expressible using the call/return architectural style that our programming languages support.

That's a lossy process.

When we can program with actual architectural connectors, auto-generating useful architectural diagrams becomes trivial.

https://objective.st

Maybe if LLMs get sufficiently advanced they can generate this stuff more automatically with some minor prompting with the code as context, but I doubt it. Not until AI can actually start understanding sentiment from code.

That's the one we're talking about here.

> A procedural diagram shows you what steps a program takes

Which is useful if the program is procedural.

> An entity diagram shows you the major high level entities you’ve selected into your system.

That's the top-level of your architectural diagram.

> I just don’t see how a single language can express all the detail.

When the program's architectural style is call/return, we can do this just fine. Using stepwise refinement we move up and down the abstraction ladder of our system and thus add/remove detail. No problem.

The problem is that most systems these days are not primarily call/return, but have to be programmed in call/return languages.

> I don’t see how S expressions solve this

I don't either, and have no idea what S expressions have to do with this.

> .... diagram generation is part of the language

It's not part of the language. But it's part of the frameworks, and when the abstractions of your language are architectural in nature, there is a very close correspondence between the code and the diagrams.

> ... discrepancies are a good teaching tool...

Yes, since the diagrams are generated, it's easy to keep older version around and generate new ones on-demand. Then you can check the differences visually, in code, or as the difference between code and diagram. Take your pick.

> Not until AI can actually start understanding sentiment from code.

Again, the point is that architecture is not actually "sentiment". It is structural. The fact that it looks like sentiment in practice is purely a side effect of our programming languages being incapable of encoding architecture in the general case.

Diagrams capture and represent abstract and/or high level concepts. Their goal is to provide a kind of mental map of how specific aspects of a project are organized in order to help developers form their mental models. This means placing the focus on key constructs and downplaying the importance or relevance of other components. There isn't an objective way to provide a one-to-one mapping between what bits of a software project are of critical importance to form mental models, and what bits are irrelevant.

> IMNSHO, the reason is that we don't have a way to express architecture in or as code.

Not true. We have more than plenty of ways to represent software architectures. That is trivial and a solved problem. So is mapping representations of software architectures to (some) source code, at least in the form of skeleton code. The problem lies in mapping software projects to a software architecture, even when the software project is clean and a textbook example of a very specific software architecture.

If the architecture were directly expressed in the code, mapping back would be trivial. It is not because we effectively compile the actual architecture in order to express it in code. Mapping back then becomes decompiling, with the added complications of architectural mismatch and manual compilation.

Diagrams are a flat map of someone's subjective and structured, interpretation of the code architecture.

Words displayed on a computer screen must be developed into supporting subjective knowledge structures.. free from the usual objective mindset of engineering.

You have to go up to first order concepts to get it. The vast majority of (popular) programming culture generates more of the same. Everybody tries to paper over this truth with metadata and it metaphysically does not work.

Computers are capable of much more, they are politically limited down to a small subset of what's possible.

You don't organize code the way you mentally model it in many projects, and nearly all languages lack a way to solve this. Annotating code is prone to the same issue as keeping a diagram up to date, and the same issue as keeping comments or documentation up to date.

The only thing I’ve seen that goes in this direction is Knuth’s literate programming. I’ve tried it. In its current form it’s still clumsy, lacks tool support and IMO doesn’t fully solve the problem of how to deal with documenting a changing piece of software yet. Knuth got his requirements correct on the first try; the rest of us aren’t so lucky.

High level diagrams should be disposable and rapid to generate. They are as important for what they omit as for what they show.

A way I addressed this was to add a checklist item automatically to PRs, “did you review and update the docs?” And put the docs in the same repo so that a code change will have documentation updates in the same PR. It’s mostly worked but still relies on discipline.

It’s kind of interesting how hard this is for some. The code change is 5 mins. Testing is 20. Documentation is another 10. I’ve seen lots of people not want to do the testing and really not want to do the documentation.

Reminds me of the army mantra, “slow is smooth. Smooth is fast.” Skipping or rushing design has never actually saved time in my experience. And I see veterans repeating this mistake over and over.

Would it not be appropriate to extend the compiler for visualising relationships between software components with zoom-in and zoom-out facilities. Zoom-in takes you to Assembly and zoom-out to the CTO.

Yes, your software may explicitly model all of these system components and potentially you can generate a system model from the code, but that would an entirely wrong approach. As a sibling comment says, this system context view describes the real world, not your software. The code is supposed to conform to the model, not the other way around. If the implementation has drifted to not be in-sync with the model, there are a few reasons this may happen:

- The legal or regulatory landscape actually changed. In this case, yes, the code may be more up-to-date and you need to change the model.

- External components your organization doesn't directly control changed. In this case, also, it may be the model that is wrong.

- The model is right and your code needs to change. Maybe you are not correctly handling an external third-party API. Maybe you're not correctly meeting your customer's needs. In the worst case, maybe you're breaking a law.

I would also think the reality at something as expansive as a bank, there is no such thing as the codebase. You don't have a single product. You have the backend data store and transaction processing system. You have kiosk software for your ATMs. You have workstation software for your tellers. You have a public-facing website for your customers. You have a mobile app. You may have an entirely separate set of insurance products, investment products, and so on. You have internal management and accounting system for generating reports. Most likely all of these need to be separate system, at least because one temporally predates others. In part because a bank is formed by mergers, acquisitions, and divestments, so some products may have originally been part of a totally separate organization and some may be destined to be their own totally separate organizations. Strategically as a company, you can't afford to give up that level of financial agility by creating hard software-level couplings between your entire product suite.

So sure, at the level of any single component, you may be able to autogenerate a high-level architecture diagram. But at the level of the entire system, you can't. This is probably most clear and obvious with something like the DODAF: https://dodcio.defense.gov/Library/DoD-Architecture-Framewor...

These are much-maligned and for good reason. They're often incomprehensible. But to the extent you're trying to model something like the operation of a war campaign, you're now involving:

- C2 systems for multiple branches of the military

- ISR systems for those same branches

- Communications systems

- Operational capabilities of all of the various intelligence agencies and foreign allies you interoperate with

- Weapons systems

- Tablet terminal, man-pack, and in-vehicle devices for your forward tactical elements

All of these are software systems, but they're developed on different cadences, by separate contractors on separate contracts, with separate fiscal appropriations bills and lines of accounting. Nonetheless, there is still a need at the strategic level to model the entire system. In order for this system to have any hope of working, it needs to be based on specifications with the expectation being that implementations will conform to the spec, not the other way around. It's more like developing the Internet than developing a web app. You can't autogenerate a diagram of the Internet, at least not one with any authority, by pointing it at the code for a server, a browser, an endpoint networking stack, and the networking stacks for various appliances like core routers, and figuring out some way to link those together, especially given you'd have to cut across arbitrarily many programming languages and code styles.

Architecture diagrams document how the software is expected to be organized. They represent the goal, not the current state. The code needs to comply with the diagram, and not the other way around.

The only scenario where it makes sense to generate diagrams from code is when we have people trying to onboard to a project that's not documented, and even then these diagrams are only generated once, polished to remove noise, and from that point onward serve as the reference.

I'm pretty sure that auto generating a diagram from some code and then trying to work out if it's semantically equivalent to something that was hand drawn is not the answer though. For one thing, the code doesn't contain or implement every single important aspect of the design.

For http API design I like to start with an openapi spec then generate as much of the server and client library implementation from this as possible.

The spec gives a language/implementation agnostic way to describe what you're intending to build that's nicely diff-able over time, and you can generate a lot of the boilerplate that's easy to screw up in a way that's both compile time (static types) and runtime (parsing/validation of inputs & outputs) safe.

I can imagine a world where a similar approach could work for higher level architecture. It's pretty common to have a shared helm chart that (largely) defines each individual logical service in k8s environments.

Taken to the extreme you could provision your data stores, and network policies etc using this approach such that an individual services chart defines exactly what it depends on. Throw in some metadata fields for descriptions and you're well on the way to having something that could generate some useful diagrams / documentation.

Of course the issue with such helm charts is that if you make them flexible enough to suit everybody eventually you'll just reimplement the underlying APIs they are calling - perhaps some approach using direct introspection of k8s resources and cloud resources with a standardized set of metadata to group and describe relationships might be more feasible.

For the moment I'll probably stick to excalidraw

Architecture is more than simply “what", it’s also "why". It binds the context to the requirements and the desired components

A diagram is not "the architecture", it’s simply a view on it, or a "projection of the model" as the c4 folk like to express it as.

I just find the idea that we should automate diagram production because diagrams are hard to keep up to date a little quaint, because you hardly ever need to update just a diagram when changing the architecture. So your actual problem is that your design documentation is hard to keep up to date, and that’s a process problem. Generating diagrams from code won’t save you there.

Developers are expected to know what they are doing and how their software project is organized.

> By always applying a lot of manual human labor?

That "manual labor" has a name: software development.

Software only changes if developers submit changes. Changes are reviewed as part of code reviews.

That sounds like a self-inflicted problem, caused by a team failing to develop and maintain their system following basic software engineering principles. I'm not sure how diagrams are relevant.

I think the best way to document a system is to write doco and just specifyc the intent of the system. What was this thing meant to do. That context is really useful for contrasting with the use of the system in a prod environment.

I'm working through a new piece of system design for a complex system using C4 with our extended teams.

In drawing the top level this time around though I realized that b/c we have a semi-disconnected mobile app (aside from a web UI) we need to degrade certain behaviors gracefully if we don't have certain results yet, either b/c we're offline, or the backend is just slow for some reason.

That rippled down into the next layer of the system as a job queue, which then rippled into the UX, because we can now say definitively which parts of the system may take an unbounded amount of time to complete a request.

I'm sure we'd have gotten to this conclusion eventually but the analysis put the issue front and center.

All because I had to draw a picture with "Remote-User --{Mobile Stuff}--> Backend."

Also whichever tool you use is up to you. I just use draw.io for example with different tabs for each layer. As we elaborate on this at some point we'll break the lower layers out to their own files.

However, I do have to credit structurizr, which outputs C4 diagrams, as a great productivity boost. One model producing multiple views at all specificities is fantastic compared to mermaid, plantuml, etc, where every diagram needs to be self contained and define and redefine the same components over and over. Branching off, editing the model, presenting ideas, has been a very successful workflow for me.

I do like the idea of describing architecture formally, but the lack of customizability of diagram output was too painful

> The C4 model was created as a way to help software development teams describe and communicate software architecture, both during up-front design sessions and when retrospectively documenting an existing codebase.

> It's a way to create maps of your code, at various levels of detail, in the same way you would use something like Google Maps to zoom in and out of an area you are interested in.

This seems different.

I have no idea why you would refer to the a code diagram as your software architecture. That's literally the code level. How is that architectural?

It's like saying the circuit diagram in the plug should go on the house blueprint. "You should use automated tools for this" ... so, it's for documenting existing code bases at the per-function level?

How is that useful for architecting / designing / planning software?

That sounds like software structure to me, not software architecture.

Sure, a map of existing software that explains how it's structured sounds cool... but I dunno. Like, if you're talking about design patterns, you're not gonna give someone a function-by-function map of how to implement a singleton. They're not stupid. You'd tell them you think it should have a singleton (or repository, or whatever).

Software design is totally a thing, and this seems entirely reasonable for designing software.

...but software design and implementation and software architecture are not the same thing and they're not done by the same people, in my experience.

This feels more like... systems design, which a software architect would contribute advice to in the way that the systems were designed so they aligned with good software architecture principals.

Maybe I'm just being pedantic. /shrug

I think this is the one mandatory part of software architecture which is giving developers information to help them make their own design decisions without constantly deferring to the authority.

But the exact place where architecture ends and development starts varies a lot.

I like to think that architecture itself decides what is important from the architecture standpoint. If architecture decides a certain low level application detail is important, then it becomes an architecture detail.

In fact, in most organisation architecture controls at the very least some top level design like components, communication patterns, APIs and technology in use.

In some organisation architecture goes as far as individual classes. Not all classes, but maybe classes modelling the domain of the problem especially, if that model is used as a language for multiple project or even implemented as a shared artifact.

I think C4 is suitable for those organisations where architecture is concerned with more low level structure than just listing applications and their integration interfaces.

Much like in the real world building of real-estate for example, I see there are different roles between the architect vs the engineers vs. the foremen.

What do you draw the line? What concerns/areas are to be covered in each of these [arch vs design vs implem]?

(also, not too pedantic)

(also, principals -> principles)

Do you mean software architecture is such as a set of design decisions? Or that software architecture is a set of design decisions?

For last few decades I have only used diagrams to design and to explain. Have found too hard to keep diagrams up to date when iterating on compiles multiple times per hour.

What I feel is needed is ways to keep code, infra and spec in sync throughout the lifecycle.

Code is too multidimensional to be approximated with 2d drawings. It just doesn't work.

What could work are more conventional architectural code patterns. Things like structured concurrency, but also for higher level. Maybe.

Floor plan has really fixed context and very limited scope. Like I don't have to explain windows ... but in software most of the time you actually have to explain "window" because every "window" or element will be different, you cannot just say this is window and copy it 20x.

How many times have you talked to someone who bounces up and down from high detail high complexity to zoomed-out simplicity in the same thought, and not understood what they wanted to tell you? We have the same issue when devs think "I will just yolo some boxes and arrows on this board, it makes sense in my head so it is a good diagram!"

Related, Archimate is also very good and there's a lot of overlap, but grounds itself more into the infrastructure bits. I've used some version of diagram approaches similar to Archimate in the past to really great effect to deconflict systems and data sources and demonstrate how all the stuff we're buying and gluing together supports specific business and user workflow needs.

Comments here about how this stuff all gets out of sync with the codebase are absolutely correct. But I've found that they can be excellent tools for use when you need to analyze a complex system or communicate systems and plans to a larger group.

It's surprising how often there isn't really any form of cohesive documentation and just sitting down for a couple weeks and drawing boxes with lines connecting them can unblock and simplify failing and frustrated efforts to move forward, especially when teams are working in the same playpen.

C4, Archimate, UML, whatever are really just good constrained diagramming languages you can use to capture specific things you wish to communicate, analyze, or memorialize. I think they aren't very good as preplanning and design tools.

Many of us suffered from the big UML push days where every facet of a system down to class definitions were supposed to be laid out in excruciating detail by some chief architect's office before a line of code was laid down. Madness like "we'll just generate the code from the diagrams" caused millions to suffer. When coding actually did start, those thousands of hours of planning usually were just ignored and the developers just did what they wanted anyways.

That being said, the biggest issue I ran into is in the diagram tooling. At the time we had Gliphy. One could not make it interactable, such that double-clicking on a node would explode the container and let someone drill-down. At best they had layers which were awfully crude to work with. And at the end of the day didn't export in SVG so were useless for embedding. Overall, a waste of time. :(

In other words, no one should change code without first validating those changes with the existing model(s).

I'll use something like this as a reference for security design as I really like its consistency.

[0] https://mermaid.js.org/syntax/c4.html

The C4 model for visualising software architecture - https://news.ycombinator.com/item?id=21032805 - Sept 2019 (59 comments)

And so wow, those diagrams in the links - how does one even manage all those boxes?!?

Architecture is easier than coding, if you stay away from more/bigger/but_my_app_is_a_special_little_snowflake mantras. Consider the age-old, tried, tested, proven architecture models of yore (other than their most excellent diagram notations, stay away from UML).

Architecture (design) is layered, not spaghetti thrown at a wall, hoping something sticks. Most (99.9%) software domains can be divided into three horizontal layers -

1. UI

2. Application/business layer

3. Data layer

(note that both internal and external service layers sit at the same level as data layers, allowing them to be called from the application layer)

Finally, there are vertical layers (shared across the initial three layers) These typically include -

4. Communication

5. Exception management

6. Security

7. Logs

Start with that as an overview/index of the entire solution, and when the beast needs to grow, just grow it. But grow it wisely. Use the standard 3-tiered architecture model. Re-use those cross-cutting concens in new applications.

Federate identities. Standardise access/permissions management.

Finally, don't feel compelled to use all the things - if the solution is services-based, don't code a UI for it. Simply connect your client to the relevant, existing service. Yes, services can be tiered, assuming your management processes are mature.

As to the management issue. Governance is what keeps your estate from turning into a hairball. Make extensive and judicious use of

1. Risk management

2. Change management

3. Stakeholder management

4. Design reviews

5. Project review (where team members grade managers)

Establish a technical design authority. The TDA concerns itself with broadly technical matters that facilitate a system-wide perspective. This might include -

1. Systems and solutions

2. Data

3. Communications

4. Quality objectives, including security

5. Development and development operations (devops)

6. Infrastructure (including but not limited to directories; networks; servers, both on-site and remote/in clouds; workstations and mobile devices).

7. Testing (Tools, techniques, platforms, languages and frameworks)

8. Locations

> As an industry, we do have the Unified Modeling Language (UML), ArchiMate and SysML, but asking whether these provide an effective way to communicate software architecture is often irrelevant because many teams have already thrown them out in favour of much simpler "boxes and lines" diagrams.

This is very true, but then they seem to have ignored the reasons why UML class diagrams were a total failure. They somehow acknowledge that nobody likes them, but then the 4th C of their model (code) is... UML class diagrams!

You don't need this. Just do ad-hoc diagrams and make them clear, with labelled arrows. Their first three diagrams are actually good examples of this.

I use Freeform now, which is super simple but works. Mark two objects and cmd-click/right click, then click "add connecting line".

Those diagrams that don't end up in presentations will be write-only anyway, and could be in any random format. No-one is going to read them.

To be fair it's still in its early of its time, I'll wait bit longer.

Just a couple of months ago I finalized yet another archictecture document, with enough class diagrams, sequence diagrams and use cases on it.

Now people straigth out of bootcamps calling themselves engineers, without having a Software Engineer degree, yeah maybe not.

Then again, they can get that book I just mentioned, and in the process maybe discover other goodies at the local library, or just ask ChatGPT.

A drawing showing the relationships between database tables seems to be comprehendible to people, until you get to over 7 tables or so, and then they don't appear to be able to make sense of it. Real databases often have magnitudes more tables than that.

You apparently have something you can do a trial with. Show it to people, give them time to digest, and politely quiz them a bit to see if it worked.

I do not believe this for a second. You're possible arguing against a straw man.

The alternative to UML isn't a "NIH UML" which uses a different array of arrows. It's labelled boxes and lines. You don't need to go to a library to understand the diagram because the diagram is self-explanatory.

Here's a really good example:

https://buck2.build/docs/concepts/concept_map/

Every arrow is labelled. No tedious textbook needed.