Every time I talk to someone who's learning to program I tell them: "Setting up your development environment will be a nightmare. I have been doing this for twenty years and it's STILL a nightmare for me every time I do it. You are not alone."
Docker is reasonably good here... except you have to install Docker, and learn to use Docker Compose, and learn enough of the abstractions that you can fix it when something breaks.
https://glitch.com/ is by far the best step forward I've seen on this problem, but it's relatively limited in terms of what you can accomplish with it.
The web is the "single top-priority" software platform, but it's in big, big trouble.
On mobile, users spend less than 7% of their time on the web. https://vimeo.com/364402896 All of the rest of their time is in native apps, where big corporations decide what you are and aren't allowed to do.
As a result, the money is going to native apps. The ad money is going there, the dev time is going there, and the mobile-web developer ecosystem is in peril.
The biggest reason people use native apps instead of mobile web apps is performance. Developers design web apps for fast desktop CPUs on fast WiFi data connections, and test their sites on top-of-the-line smartphones costing 5x-10x as much as the cheap smartphones people actually carry around.
Web developers have to solve this performance problem the way we've always solved our problems: with a new framework. ;-)
But specifically we need a framework designed to generate HTML with no JS at all, and designed to run in a Service Worker, which is a little web server that runs directly on the user's phone.
This style of app is often called a "Progressive Web App," and there are plenty of frameworks that support PWAs, but they generate PWAs on top of a single-page app framework that downloads megabytes of JavaScript running on the main thread. PWA is an afterthought for most frameworks, but we need it to be the centerpiece of the design.
No amount of processing power matters if you don't have the data.
Everyone in the industry focuses too much on the processing side: objects, functions, containers, VMs, k8s, etc. but nobody really gives proper attention to data, its provenance, where it stays and where it goes, etc. I'm not saying engineers don't think about these things, they obviously have to think about it at some point. It's just that data is always accessory to the story. It's like processing is the cool kid and data is the stinky one nobody wants to approach unless you have to. Look at the 12 factor principles for example, where is data in there? How easy is it to take data from one place/cloud/database to another? Data is the raw material, it needs to be the primary concern in programming languages and architectures, not objects or functions, containers or whatever, those come after, not first.
Every website and app and network service seems to be reinventing the wheel here, and end up creating little silos of trust data, whereas in principle it should be possible to receive a (locally) consistent answer to the question of "Is this person in good standing with the other humans they interact with?" whether that person is sending you an email, or writing a software library you are downloading, or creating an account on your website, or selling you goods on Ebay, or offering you a lift through a ride sharing app.
Most of the time the users haven't taken enough time to understand their own problems and have trouble articulating them in a way that's meaningful for a product manager or developer. And on the opposite side of that product managers and developers often do not develop sufficient domain experience to understand the problems users are trying to express.
This is why the absolute best software is built by people who are developing for themselves. You know when it's not solving your problem and you fix it because you know nobody else will.
That seems nuts to me. I'm not sure how you fix it without being Amazon/Google. People moan about Terraform too before that gets mentioned.
https://github.com/frankmcsherry/differential-dataflow
Lots of very active CS research in this area though.
Far too often when I want to learn something new I either find there is not much documentation or find that there is a large mass of documentation that probably has everything I need but is so disorganized that I can't figure out how to approach it.
Another documentation problem, especially with open source projects, is that development and documentation are often loosely coupled, if at all. The people doing documentation usually don't have the resources to keep up with development, and so even if there is good well organized comprehensive documentation it is usually obsolete.
Something like Ubuntu Edge for those who remember it, but with more local storage -- TBs maybe -- more connectivity out of the box.
I want to be able to plug APIs together, process user input, have persistence and identity, without writing so much boilerplate.
Being able to update libraries, tools etc. automatically and without friction. Right now upgrading is so tedious, error prone and painful that most places just keep using ancient versions that are not only lacking bug fixes and newer features but are a huge attack surface.
WASM is a great advance over what came before in many ways, but still has some room for improvement. These are all problems with known good solutions, but there is no holistic platform that integrates these smoothly. It's largely a hodge podge of various programming languages, configuration systems, build systems, scripts, etc. Look elsewhere in this thread for people complaining about how difficult it still is to setup a development environment. Embedded programming is typically even worse, though it's improved dramatically since the rise of Arduino.
It needn't be this way though. A well designed programming language can be used for configuration, scripting and more. Racket is a good example on the dynamically typed side, and F# is a good example for a statically typed language along these lines.
I am willing to go out on a limb and say that as much as 25% of software engineering time worldwide is wasted due to poor documentation.
It's an asymmetric problem too. If someone benevolently funds a team of engineers for a couple of months to write great docs (with detailed examples) for top 500 libraries, frameworks, APIs. They could increase global productivity of software engineers by %25 percent.
And I know there are lots of html / web based things you can run on desktop but they are even more complex, for me at least.
User interface responsiveness.
The Web being based on standards that leak users' data left and right.
Thus, on a longer term, you might want to identify the cycles and look into opportunities beyond the current phase in these cycles, and whether there are under-utilized ideas there.
For instance, for unix-style command line operation, we have the idea of piping data between applications and combining multiple applications to perform a job.
These applications communicate through very simple protocol, the text file format, where one line means one thing. Thus, if we want to combine more complex applications, such as operations on image files, each needs to implement their own processing for various file formats etc.
My idea would be to try to increase the abstraction level of operating system from files to something more generic.
For example, what kind of things I could script more easily if the operating system would allow me to read source code tokens/statements/packages in any language? Or images as an abstraction regardless of their file type?
Cookie-cutter developers using open source they don't understand to implement mission critical infrastructure for companies that don't understand the internet. Its a recipe for disaster.
Google died when they stopped being well informed librarians and started being aggressive salespeople. It's time for a new search engine to step in specifically catered to the curious.
In theory: Prove that one-way functions don't exist, or explicitly construct one. Similarly, prove that P!=NP, or similarly settle the question.
Having to check out the branch, install dependencies, build the code and run it only to then go back to the UI to see the changes and comments is very time consuming and being able to test suggested changes instantly would save me a ton of time.
I would gladly pay for this.
There is still room for Domain Specific languages like sql, html and such but for imperative programming there are way too many options all filling almost the same need with just slight variation. Even with the rise of all types of VMs and cross compilers we are still porting mountains of code to another language just because the execution environment was slightly different. The gap between embedded and web is also too big, for embedded you are stuck with prehistoric c++ with dangerous syntax, hour-long build times and days of figuring out how to correctly compile and link that shared library. vs web where your only options are dynamic languages where both safety, predictability and performance are a joke. I also realize safety and performance vs productivity often contradicts each other but not as much as often argued, there has to be a better middle ground than what we are stuck with today.
In isolation, this should be an achievable task, especially if dropping any legacy compatibility. What might be difficult is is people want legacy compatibility and proven in use, so adaptation-rate will turn this into "there are now 15 competing standards" and a "peace on earth"-type problem.
A large and increasing amount of human-made energy goes to computation, yet only a tiny proportion of software is written with energy efficiency in mind. Most programmers don't even have the tools to answer how much power their programs consume.
At a start, no compute benchmark that measures wall duration should ever be taken seriously if it doesn't also include energy consumption.
Java applets came 20 years too early - potentially had the power to do everything we do with the web today but 10x faster and more cleanly.
The web remains one place where you lack the freedom to easily use whatever language you like - WASM will end the era of JS if they do it right.
The JS ecosystem is extremely wild and turbulent - even something as simple as "I need this project to be built exactly as it was in August 2017" is almost impossible with the npm world.
Meanwhile native apps compiled in 1985 still run, and can even build today with minimal fuss.
Lets be honest - how many of you use JS because there was no other option? Its not a terrible language - in fact I like JS/ES7 more than python, but it's still one of the pillars of chaos in the world of programming
If you could solve this problem, you could convince management why engineers need offices, for one.
We should not be rewriting and adopting a slightly better but incompatible version of make every five years, or waffling between SQL and NoSQL, or churning back and forth between slightly different versions of MVC paradigms. Or going from tables to div-tables to flow layouts to css-grid.
I can think of 2 ripe opportunities, that I reference often.
1. Comments integrated with code as associated records that can overlap, which provides context, not merely inlined comments. The state of understanding code is horrendously inefficient by design.
2. Put a more type safe language on the browser with concurrency primitives. Promises are a hack improvement and Python is too rigid in ideology to make it eventially compatible.
1) computers handle bits and bytes, not information. If computers can be made to create, search, update and delete pieces of information, instead of bits and bytes, 90% of code would go away and life would be much easier for all of us.
2) programming is done wrongly and poorly: we write a program to do a specific job, without any proofs, with serial control flow, we compile it, we setup an environment for it, etc. Instead, we should write hierarchies of programs, each level of hierarchy should have its own proofs (i.e. the specifications should be part of our programs), control flow should be event based, programs should be running as soon as we write them in a live test environment etc.
In other words, forget files, processes, handles, databases, source files, bits, bytes, the command line, UIs etc. All these provide some level of abstraction that doesn't really scale to what we actually need. We need another level of abstraction: the piece of information.
Which should eventually include a piece of code that communicates with the outside world via events, and that code would be composed of other pieces of information, would be fully creatable, searchable, updatable and deletable just like any other sort of piece of information.
And UIs should be creatable, searchable, updatable and deletable pieces of information as well.
And a global communication language would replace all command line interfaces, UIs, and programming languages: we shall talk to our UIs with this language, and the UIs shall talk to us by using that language as well, using graphical representation when needed.
There are too many un-opinionated choices today, so people waste too much time on choices for the various pieces. The whole ecosystem is very fragmented.
A relatively sane, top-to-bottom framework that isn't the fastest, or "best", but "good enough"...would be welcome. Maybe Golang or Rust or Node will progress to having their own "Rails"?
Not trivial but maybe automated by some meta-meta-build tool?
Cannot tell you how much time I spend fighting solved-since-1980 RDBMS problems at the application layer. But we really are too big for single-master (pushed it to its breaking point and a little beyond).
Web design seems to be unifying around progressive web design principles and too many devs spend way too much time recreating the same variations of CRUD apps.
If any new web feature could be distilled to a metacode base and then restructured from there we could maybe find a way to escape this nightmarish hydra of web technologies.
2. Tools to help making parallel programming easier, without massive impacts on efficiency. Future CPUs are offering more performance primarily through more cores and wider vector instructions. Currently to use these efficiently is often very hard.
3. A replacement for C/C++. Rust is promising, you could contribute there, or to Zig, or come up with something better.
4. large scale, high quality studies into the efficacy of various programming ideas, languages, methodologies, etc. (Does dynamic typing increasing or decrease productivity, or under what domains does it do one or the other, does functional programming reduce error rates, by how much, and at what performance cost? etc
In a perfect world, I'd like to run an analyzer on my CRUD app's source code, and have it list the 37 vulnerabilities I overlooked.
Probably not a candidate for "most" but maybe worth mentioning: pgp
There's also lots of non-technical debt; indeed not even any of the above are purely technical, at the least getting solutions adopted likely requires a lot more than just serious hacking.
Surely there are existing projects, or at least communities, that have formed around taking on what participants thought/think of the most importance piece of technical debt, at least that they were/are equipped to take on; perhaps [non-]reproducible builds, shotgun parsers (langsec)?
I'd, instead, follow the Alan Kay idea of antecipating 10 years in the future with some new hardware that allows a new computing platform and then see how I can capitalize in it in some kind of pure way (removing as much technicalities as possible, like Self, LISP or Smalltalk did).
For inspiration on what form it could take, I'd take a look at all Sci-Fi I can but pay special attention to Bret Victor's: Humane Representation of Thought https://vimeo.com/115154289
Specify a problem statement and let the computer figure out a way to implement it but take this idea and apply it to SaaS Applications. Here's how a domain model looks like, go figure out a database-backed API application implementation for me.
Because this affects cognitive health and productivity of just about every tech employee, it truly is the biggest problem and is no overstatement at all that we would cure more diseases, feed more hungry people, develop better policies for people in need, or even just satisfy consumer demands more profitably, if the scourge of open-plan offices finally goes away.
Very few other problems faced by engineering workers are so vast and systemic as open-plan offices, with such far reaching value if it is solved.
https://testing.googleblog.com/2007/01/introducing-testing-o...
There's nothing out there for this.
I'm talking about things like sign-up workflows, password recovery, two-factor authentication, role-based access control, search, approvals by manager, integration with BI tools etc.
There are application development frameworks that solve some of these problems for you, but they tend to be so heavy-weight and/or impose so many restrictions on you that they hardly seem worth using.
The root of the problem is that data denormalization is broken [0]. The fix is doable in theory, yet not done yet, and not talked about enough.
[0] https://medium.com/@lironshapira/data-denormalization-is-bro...
Right now there is no differentiation between actual engineers who write original code and button pressers that either live in configuration hell or that mindlessly need design patterns to tell them what to do.
Programming is the act of writing instructions and yet so many developers can neither communicate in writing nor plan a series on instructions.
In other words: solve the Data Model Coupling problem programmatically. Applied to an RDBMS, one could imagine a graph with weighted edges representing coupling between database tables and dynamically reverse-engineering joins into DB-isolated API calls.
Once you get to a point where you can demonstrate that 'doing it faster' is not an option, everyone gets better for it.
Right now, all you get is bypassable CI pipelines, crappy in-IDE tools that sometimes work, and very costly static analysers that usually don't work and are more designed to please the clipboard warriors.
Emphasis on exchanging information between 2 unknown programs.
So much programming plumbing (parsing text etc.) to do just that.
So many hours wasted chasing API documentation to figure out how to call an incantation.
So many software features hidden away and inacessible unless used through a UI maze.
So much code sitting there unused and undiscoverable.
(That and the 800lb gorilla in the room: programmers are fashion-driven to the point of absurdity.)
Except, thinking about it, I don't think it fits your description because it can't be solved by the methods you describe. We are "complexity junkies". Most of us haven't hit rock bottom, don't see a problem, or are well-paid to feed our habit.
We have tools and concepts that would do the trick, but we ignore them.
Consider Elm lang. Takes all the complexity out of writing web apps, has a history of zero bugs in the code it generates, doesn't get traction because...?
- - - -
Dr. Margaret Hamilton (of Apollo 11 fame, who coined the phrase "software engineering") developed a system of software construction she called "Higher-Order Software" that eliminates the sources of most programming bugs. Sadly, it was critically panned and has languished in obscurity for decades. See "System Design from Provably Correct Constructs" for more info.
- - - -
Graydon Hoare gave a talk on the history of compilers[1] and he doesn't mention Prolog once. Is it possible he doesn't know about the research into logic programming and compilers?
E.g. "Parsing and Compiling Using Prolog" Jacques Cohen and Tim Hickey ACM Transactions on Programming Languages and Systems 9(2):125-163 · April 1987 DOI: 10.1145/22719.22946 · Source: DBLP https://www.researchgate.net/publication/220404296_Parsing_a...
1. Introduction
2. Parsing
2.1 Bottom-Up
2.2 Top-Down
2.3 Recursive Descent
3. Syntax-Directed Translation
4. M-Grammars and DCGs
5. Grammar Properties
6. Lexical Scanners And Parser Generation
7. Code Generation
7.1 Generating Code from Polish
7.2 Generating Code from Trees
7.3 A Machine-Independent Algorithm for Code Generation
7.4 Code Generation from a Labelled Tree
8. Optimizations
8.1 Compile-Time Evaluation
8.2 Peephole Optimization
9. Using Proposed Extension
10. Final Remarks
Long story short, if you want to write a compiler it's easier and faster to learn Prolog and write it in that than to learn to write a compiler in whatever lower-level language you might already know.
[1] https://thenewstack.io/rust-creator-graydon-hoare-recounts-t...
Already solved, but not used.
2. False advertising, overhype.
Related to 1)
I truly miss many immersive Flash experiences on the web.
In my opinion there are two big angles here. One is memory safety, which is the primary cause of remotely exploitable vulnerabilities these days (see https://twitter.com/LazyFishBarrel for some stats). Any evidence-based approach that reliably reduces memory unsafety bugs is productive - whether that's "rewrite it in Rust," "rewrite it in Go," "rewrite it in Python," "rewrite it in Java," "write really good C static analysis tools," etc.
The other is getting software updates into people's hands for a reasonable price. If you buy a cheap Android phone, you're buying an insecure Android phone. There are few options for a cheap iOS phone, especially one that's still receiving security updates.
See https://googleprojectzero.blogspot.com/2019/11/bad-binder-an... for a good analysis of a combination of these two problems, specifically weaponized in the wild by NSO Group. One part is that it's a use-after-free. The other part is that it was fixed in Linux, and it didn't make it into Linux for two years.
If I had a large team of serious hackers at my disposal to solve problems for the world, I would work with some of them to fix the highest-risk code written in memory-unsafe languages and get the fixes upstream, and I would work with the rest of them to fix the various process problems that make it hard for Android vendors to upgrade to new versions of Linux and other components continually.
(Note that both of these problems are really best solved by a team that can continue working on the problem indefinitely, not a strike team that delivers a thing and then declares the job done.)
As it stands, both beginners and experts have difficulties understanding exactly what their own programs are doing as well as what programs written by other people are doing. We encode complex algorithms in static, abstract text descriptions that are hard for humans to understand and reason about. We have to imagine the behavior in our heads from these illegible descriptions. The behavior and internal workings of their programs are invisible.
Not to mention that when trying to modify other's programs there is often a huge communication problem, trying to construct a mental model of what the program does is a tedious and often impossible endeavor because of missing context. Just think, how is it even possible to make "write-only code", code that was understood when written but is now completely unintelligible. To me, that should be impossible. Or think about how open source software is open in that its code is available, but closed in terms of being easily understood — there's the formidable cognitive challenge of understanding the program well enough to be able to modify it to one's ends. For most people, this is a significant and unreasonable effort.
What to do about all this? To me, the answer is redesign programming so that it is primarily about communicating behavior to humans. A couple things I've made toward that end:
* Legible Mathematics, an essay about the UI design of understandable arithmetic: * FuzzySet: interactive documentation of a JS library, which has helped fix real bugs: * Flowsheets V2: a prototype programming environment where you see real data as you program instead of imagining it in your head: * REPLugger: a live REPL + debugger designed for getting immediate feedback when working in large programs: * Marilyn Maloney: an interactive explanation of a program designed so that even children could easily understand how it works:In general, I think it's a neat research direction to redesign many concrete programs by hand in the most understandable way possible (using custom graphics, interactivity, game design mechanics — all the best things we have for helping someone understand things through media), and then use those experiments to work backward toward programming languages/environments.
In the end, programming is essentially only limited by human understanding, so that's the most significant engineering program out there.
One example: lots of software devs have no training and/or interest in security, and employers have no way to vet that.
Another: there is no way for users to trust SaaS security. I have no idea whether (as a random example) Atlassian has a great security culture or a terrible one. We just trust people who seem trustworthy, usually due to their polished marketing.