Software design patterns for projects without a scope?

I personally think that you’re doing the right thing. Thinking too much about extensibility too early or applying patterns often makes your code overengineered, and in no way can guarantee that it can be extended in the right direction.

Having to rewrite existing features differently is mostly a consequence of you not being an expert in the business application you’re writing.

Maybe if you’d invest more time upfront to understand deeply what the application is for and how people are going to use it, you’d be able to spare a few mistakes.

But even being an expert and talking to people doesn’t shield you from that problem. There is always a new feature that contradicts how the code works. It’s just a consequence of software being so malleable.

As a car driver, you can’t ask the car maker: “hey can the needle on the speedometer turn red when I go over 60?”, or “hey I need this car to work with both gas and diesel?”

This would be perfectly reasonable in a software equivalent world.

In the end, you just need to explain this to your client…

I do much the same kind of development.

Document the design choices you made and why.

That's it. That's your answer. If anything, spend even less time worrying about 'proper' abstractions. As you've seen, you'll wind up guessing wrong and making more work for yourself/other future maintainer as often as not. Keep it simple.

At some nebulous future endpoint, if a client reaches a growth stage where they are ready to take the next step beyond small-scale prototyping, great! It's a good problem to have. Ignore the code you've written and instead read the history of your design decisions that got you to where you are. Business workflow will have co-evolved, and some of your decisions would now flow differently if re-evaluated. Write down the changes, and there's your rough draft spec sheet for version 2.0.

You're doing it right. Consider how truly difficult the environment is you're constrained to operate.

None of the users know what they really want, because the end users don't know what's possible, nor how difficult any particular feature is, so they're guessing in the dark.

You don't have the years of experience they have at working their jobs, and knowing what is quite easy or quite hard, in the same manner as the above paragraph.

In spite of a heavily information poor environment, you and the users work together to build systems that meet their needs, and they are happy.

Code is the crystalized knowledge of the program domain gathered over time. You're correcting that as you go, rewriting as appropriate. It sounds to me like you're doing refactoring as appropriate. Using the word refactoring should help you with the search engines quite a bit.

Looking back on things, it seems obvious what the right way to do something is, because you now have far more hard won knowledge you lacked back then.

Again, don't judge yourself harshly. It sounds like you're going fine.

Sounds ok to me. Probably the most flexible architecture I worked on (back in the waterfall-ish quarterly delivery schedule days) largely consisted of a rules engine built in an Oracle DB with a JSF front end that would dynamically build the display based on the fields present on the work item. This meant we could elevate things like field changes and some business logic changes without elevating the monolith just by making Oracle changes.

It really worked pretty well. The problem was, the business got used to the easy/fast changes. Then they wanted to make some larger changes involving integrations with a bunch of other systems. To do it right we would have needed rewrite/rewrite a large part of the system. The system could have used some modernizing anyways. Nobody (in management) liked my opinion. Well, 2 years after leaving that team they hired another company to rewrite and modernize the system... I guess they took my suggestion after all.

Personally, I think our existing languages and tools are still too primitive to avoid rewriting considerable amounts of code as requirements change. At some level, rewriting is inevitable. We have developed some techniques for minimizing it. The shift from assembly to C reduced the amount of rewriting necessary when the underlying ISA changes, for example. But rewriting sections of code still seems necessary quite often, regardless of paradigm.

As far as I can tell, the best technique is to write straight-line code as much as possible, and then hoist out (ideally pure) helper functions when you notice clear duplication. Then, if and when you need to rewrite, it should be easy to understand what needs to change, and you can easily inline and then re-hoist helper functions as needed.

If you haven't read it already, I highly recommend this post about semantic compression by Casey Muratori: https://caseymuratori.com/blog_0015

The most important aspect for me is getting the strategic boundaries right; the way a company is organized might not be the most optimal, so getting deep into the business and data, and at least trying to get the high-level architecture right before starting your first line of code would be my best advice.

After that, I tend to make a distinction between inherent complexity and accidental complexity.

If it has inherent complexity, I tend to spend extra time at the whiteboard to make sure I have the best possible model based on the info I have at that point in time, and I also refactor relentlessly.

For everything else, I try to make the code easy to throw away and replace. Typically things start out simple and grow more complex after a while. Trying to retrofit a proper model usually costs more effort than a rewrite.

For me, spending over a decade deep in the domain-driven design (DDD) community has proven to be the best investment I ever made in that area. However, as DDD is getting more popular, I noticed the emergence of echo chambers and some "beginner expert" thought leaders who are starting to teach before truly understanding DDD, and others relentlessly productizing and pushing their offerings, so make sure you only engage with experts who really care. (A good heuristic to detect a beginner expert tends to be someone who uses too much DDD lingo, but YMMV)

> oftentimes I'm rewriting existing chunks of code to incorporate the new features because I chose a certain structure that no longer makes sense.

You're doing it right.

> the client usually doesn't know what comes next until they need it.

Do you know what comes next? If not then designing for requirements that may not exist is likely to take your code in the wrong direction.

What I learned from 20 years of development experience is to follow the below in that specific order.

1. YAGNI - do not try to solve a problem which is not yours

2. SOLID - try to build in an extensible way, so you can later refactor it easily

3. DRY - it is time to modularize when you start copy/pasting

So, I can say that you are on the right track.

Prediction is very difficult, especially if it's about the future. YAGNI.

It feels stupid to not have any pattern or theory or code reuse, but when there is insufficient data, the null hypothesis is the best guess.

I think: have confidence that you're doing the right thing. Write the code in the most obvious way to do what's needed - the least theory, model, pattern. That will be easy to read - as documentation for later.

Eventually, you may have seen enough of the problem (or part of it) to be confident you understand it. Then you can throw away and rewrite that part. Probably, even trying to adapt the old code is not worth it - more efficient to start fresh.

Give a bit more context on what you are already using so we are not shooting in the dark.

You are likely already using these:

With a typed language the compiler helps you refactor more efficiently.

No spaghetti code, instead move logic into coherent (ideally functional) libraries.

I’ve found the functional object-oriented style to give most flexibility for changes. (Ie. Animal.kill() doesn’t modify it but returns a new Animal object.) Copy-on-write everything, use appropriate data structures to make it efficient.

Simplicity. Build granular parts that can be assembled in various ways. This way, when there's a seismic shift in requirements, you can hopefully just rearrange some of the parts instead of starting over from scratch. Beaches aren't really affected by earthquakes.

Overarching frameworks and patterns work well when you know ahead of time what you're building; unopinionated granules work well when you don't.

Fundamentally I think you are doing it right, especially because you are not afraid to rewrite substantial parts of your code, while being pragmatic about what that entails.

Never forget: we also need to feel comfortable in a code base. our happiness matters, it's not just about agility. A tidy, continuously improving code base has more long term effects than just "shipping stuff quickly".

To dive deeper into the topic I can very much recommend two books:

A Philosophy of Software Design. Very practical, friendly book, actionable advice and some sound mental models to apply to designing interfaces, error handling, modules and so on.

Software Design for Flexibility. Spiritual successor of SICP. Very challenging, dense and in some ways radical. Typical Sussman-style of throwing challenging concepts at you in rapid succession in a logically sound and clear language. Full of exercises, questions and interesting references.

Can you give an example or two of the software design patterns you’ve found helpful in the past? I had a big comment typed out but then realized there are many kinds of, like, “scopes” in your post. Are you looking for abstractions to cut down on your time-to-prototype? To extract a “framework” from your extant software to increase your efficiency over time? Something else?

You want agile and test driven development.

Once you have a working prototype you collect user stories to modify the prototype. Clean up your code and move it towards solid principals using TDD.

As others have said, I think it is not good to prematurely optimize a solution with too many unknowns.

I think a better approach would be to ask the client about what parts of the system are likely to change. If they are not good at thinking about this in a general sense, you should ask that in a more specific sense.

If you can get a better sense of what will change, it guide a little bit more of the design.

you just described 99.99% of software projects. There is no magic bullets. Keep the iteration going and don't overengineer.