What are interesting new developments in databases related fields?

A few DBMS innovations:

- The "No[t just ]SQL" movement brought us schema-free DBs (among other things; note that I personally prefer schema enforcement and strong typing);

- CouchDB and others started to integrate MapReduce-style support for distributed processing;

- Increased support for polygon/centroid indexing: PostgreSQL -> PostGIS, SQLite -> SpatiaLite

- Graph databases: RedisGraph, SparkSee, Neo4j, AllegroGraph and others - as well as Cypher as a de-facto query language standard (supported by Neo4J, RedisGraph and others)

Some senior DB people like Gerhard Weikum ( http://people.mpi-inf.mpg.de/~weikum/ ) switched from databases to Web mining, because they believe that is where the action is now: acquiring/learning structured data from unstructured sources.

So, I'm an independent researcher in this space and the founder of https://www.adama-platform.com/ which, at core, is a new type of database. The way to think of it is as Key to Database service where the Database is an atomic unit like a user, document, application state, or game. I haven't tested how big a database can get, but I expect to be able to get towards 1/2 memory.

I'm moving slow as I'm building it myself, and I don't even have joins yet in the language (although, I'm not sure I need them).

I believe current databases represent many mistakes. For example, how we think about privacy is a cluster fuck. The pattern of having a single root password to a database then exposes your data to anyone with network access. My opinion is that privacy, authentication, and access controls should be properties of the database layer.

A long term goal of mine is to build a nice layer for interacting with large documents which provide great direct product experiences, and then offload the larger scale aspects (i.e. cross document search) to dedicated indices or query engines. By focusing how product interaction happen and isolating to a user or document means fantastic control over privacy, audits, latency, and compliance.

Serverless DBaaS providers and HTTP APIs are a huge new development. Mongo Atlas Serverless, Planetscale (MySQL/Vitess https://planetscale.com), Upstash (Redis, https://upstash.com) and Neon (upcoming Postgres). They add features like merge requests on DDL and make managing DBs as easy as managing Lambda functions. This reduces maintenance overhead significantly.

I am building a whole new kind of general purpose data management system that does a bunch of file system and database functions. It is currently in open beta (www.didgets.com) and anyone can download and try it out in just a few minutes.

It can't do everything yet that a full-functioned database like PostgreSQL, MS SQL Server, or MySql can do, but it is exceptionally fast and has a bunch of unique features. My benchmarks for hundreds of queries against some pretty big tables (e.g. 100M rows, 100+ columns) show it can be up to 10x faster.

It is a columnar store with a flexible schema so it can easily handle Json data where a single column is an array of multiple values. It creates a 3D table where each column/row intersection can have multiple levels. It is really easy to drill down to do analytics against subsets of a big table (e.g. only customers in California).

It also accomplishes its query speeds without needing separate indexes against any column(s). You get blazing speeds after creating a table and importing its data no matter what column values you query against.

The website (www.Didgets.com) has some short demo videos that display what it can do and how fast it runs.

There’s a bunch of interesting areas. The ones I’m focused on right now:

Incremental View Maintenance

Materialised Views

Multiple indexed databases that allow for row, column, and search accesses very fast

Real-time replication

What I'm excited about are the recent work by the likes of Hasura, EdgeDB and Supabase to provide an API - whether GraphQL or Rest - to work with Postgres databases. MongoDB has many many faults, but one of the things it did better than SQL databases was that its API was a lot easier to use from application code than SQL. I'm glad to see some activity in the Postgres space to address that.

Database as a Service (DBaaS) is probably the one that will affect most developers. Aurora, PlanetScale, Cockroach, MongoDB Atlas, Yugabyte, ClickHouse Cloud, Materialized, Timescale... there are dozens of fully managed dbs now that abstract away some of the most difficult problems of managing your own clusters.

On the more technical side, I agree with brad0's list, especially incremental view maintenance.

Doing more math in databases. eg. PostGIS, vector math, ML type workloads supported by declarative SQL.

I’m personally very excited for developments towards Incremental View Maintenance (IVM) in Postgres.

https://wiki.postgresql.org/wiki/Incremental_View_Maintenanc...

https://www.youtube.com/c/CMUDatabaseGroup

Self driving, autonomous databases are an interesting study

Practical?

Navigating the wave of AWS/Heroku etc replacements.

Fly.io, Netlify, etc

Ottertune