I'd love to hear about what interesting problems (technically or otherwise) you're working on -- and if you're willing to share more, I'm curious how you ended up working on them. This time, with a twist. Please include web technologies too if it's really niche and not a lot of people know about it
For about 8-9 years now my goal has been to build a cell from scratch for less than $1000 and enable anyone to be able to do so. While I’m handling the other bits elsewhere, DNA synthesis is a key technology primed for open source disruption.
Body doubling is known within the ADHD community and entails performing a task in the presence of another. More details here: https://www.healthline.com/health/adhd/body-double-adhd
It helps to engage motivation by using another person as the proxy. I wrote a bit about how I think it works here: https://doubleapp.xyz/blog/body-doubling-proxy
The technique goes way beyond just ADHD applications for executive functions and is something we tend to do anyways, e.g., running with a friend, studying in a group, etc.
It solves an issue for myself and I truly want to help others with the approach by making a way to stay accountable through the help of others.
I guess it's pretty common practice for corporations or private equity to snatch up smaller independent practices and then run them like a sweat shop. Her current CEO's resume is a list of Starbucks, Walmart, and a few other corporate roles that are totally unrelated to the medical industry.
We're tinkering with a platform that can somehow disrupt this cycle and add some transparency to the practice structure for the would-be employee.
The current form is kind of a niche Glassdoor, or we've been thinking of it as the "online dating between veterinarians & practices." Still playing with the concept.
Anyone have any thoughts on how to take out these massive PE firms that are plaguing the Vet Med industry?
Here's some more info on the problem: https://www.ftc.gov/news-events/news/press-releases/2022/06/...
(Backstory: Started a Twitter account in July (twitter.com/@betterstreetsai) posting DALL-E-generated street transformations, it immediately took off, got lots of press, etc. and made me realize how huge the demand for this stuff is.)
Feel free to send me a DM on Twitter if you're interested in helping!
Search itself is a fractal of interesting problems. Haven't had much time to write about it lately, but I've pretty much doubled the size of the index and re-written a lot of the query logic to make it much better, faster, and more accurate. Will do a write-up eventually, since it may be relatively explainable without getting too into the weeds that the audience dwindles entirely.
I keep having breakthroughs that make it in one way or another better, but as soon as I do I find something new that could be improved.
Kinda bonkers it's been possible to build this alone and run the entire thing on what amounts to a souped up PC :P
You would think that this is a well established field. But then I found claims in online discussions (including at the W3C) that were obviously wrong on a mathematical level. The more I looked, the more the arguments fell apart. So instead of reading online discussions I concentrated on reading the actual formulas as well as research on the topic. I now belive that the reasons people cite for choosing one algorithm over another are often completely bogus.
I tried to bring this up at the W3C, but the response was anything but friendly. Not sure what I did wrong. I still think that I have something valuable to contribute, but I am not sure how.
If you are interested in more details: http://tobib.spline.de/xi/posts/2022-09-10-contrast-algorith...
To put it simply, I think the trend towards no-code/low-code is misguided. The assumption behind these products is that code is slow, difficult, and expensive. I disagree - I think that "code" is merely a formalized written expression of what you want. It's actually the most efficient, easiest, and cheapest means to solving digital problems.
I believe anyone can write code if it's specific enough to a domain that they understand, so what I'm trying to do is create a hyper-domain-specific language around designing UI components. The goal is to have a platform agnostic base syntax, with "dialects" that can extend the language for any current or future platforms.
The idea is that designers could write in this language, and developers could build tools that will transpile this code into a consumable format for whatever system they're building for.
I have a very rudimentary demo site that's like halfway built: https://matry.design
edit - here is a simple example of what a component might look like in this language:
component Button
variants
color bg: #007BFF
text label: Click Me
elements
shape ButtonContainer
text ButtonLabel
style ButtonContainer
fill: $bg
padding: 10px
style ButtonLabel
content: $label
font-size: 16px
The hardest part of it is actually deciding what do I want from the time I spend working. Many of the folks I know who do interesting work are either woefully underpaid or chronically overworked.
I'm hoping I can thread the needle on that one.
The structure it currently takes - academic journals - made perfect sense in the 1600s when that structure was developed, and it continued to work reasonably well for distributing academic results up into the early 1900s. But then it got privatized. Now there are tens of thousands of academic journals, 80% of them charge a fee for access, and most of the remainder charge a fee to publish. Often thousands of dollars.
Given that science works in the aggregate and you can't know if you have the real answer to your question until you've accessed all of the literature on the topic, this structure is now making it impossible for people not in the institution to even figure out what we know on a topic. And hard even for people in the institution.
The ultimate decider of policy in a democracy is the average citizen. The people who decide our government policy don't have access to the primary source material of science. In the US, a lot of really important policy is set at the municipal level (city, county), and in that case the people actually writing and implementing the policy also don't have access.
If we can develop a web platform that does all the journals do (match papers to qualified reviewers, maintain literature integrity by filtering out bad work, and dole out reputation), then I think it's possible we could draw publishing out of the journals and into the open where it belongs.
I've got an idea for a platform that does all that by crowdsourcing it (the journals are already crowdsourcing it - just manually using an editor). It's basically Github+StackExchange for academic publishing. It works by tying the reputation system to academic fields, so papers are tagged with fields and then reputation is gained and lost in those fields.
I'm building it now, I'm month or two out from beginning a closed beta. (Aiming for end of October, beginning of November.)
I wrote up a detailed description here: https://blog.peer-review.io/we-might-have-a-way-to-fix-scien...
Why aren't there more employee-owned companies? And I don't mean solely shareholder programs, I mean actually owned by every employee and they are paid dividends of profit after all planned R&D costs. Similar to the Alaska oil pipeline bonus. Even interns.
All studies show that the companies structured this way that do exist (they seem few) have much higher output, quality, and happiness among staff.
After more research I found that this exists: https://esca.us/ - Wawa is a member, pretty cool.
Talking more to others about the idea I've heard interesting stories. Some machine tooling shops have an actual employee-owned setup. All employees are incentivized to make every product output great and keep profits high, because they all share the profits.
Anyway, I'm not actively working on this. But wanting to shape this idea more in the future. It feels like it could help the current state of America. But perhaps I'm too hopeful and naive :-)
---
Hobo Johnson - My Therapist:
"The idea's about equity, it's about wealth
Most think that it's dumb, you should think for yourself
If I buy a pizza place that makes a definite profit
Yeah, let's say yearly, the owners make $100,000 off it
And if I buy this pizza place for, let's say, $300k
And when the workers recoup in three years, I'll sign it over that day
And wouldn't everybody not see
They should buy their pies from me?
You'd rather have a boss
When you can work democratically?"
---
"Incentives are the strongest force in the world. They explain why good people do awful things, why smart people do stupid things, and why ordinary people do amazing things."
- Quote I have pinned from @morganhousel
We're not building a huge machine to try to suck the atmosphere through a straw; we're doing chemistry in the open atmosphere. Plan to have a pilot running within the next year and within five years be removing up to 100 Mt per year.
Had a cycling accident years ago, became much more sitting intolerant afterward, probably made me quit coding as a career.
Have been testing prototypes for a year, people have done hours long stretches on a 2-3 min interval feeling no pain or stiffness at the end. A study showed the same strong results.
Website: https://www.movably.com/ Early user review: https://www.youtube.com/watch?v=hUjkbMc_xBw&t=1s
The architecture allows any value to be mapped to one or more keys and any key to be mapped to one or more values (unless an attribute on the data object prevents it). These KV stores can be used to attach tags to other objects. They can be used to form columns in relational tables (i.e. a columnar store). It can be used to create indexes into file contents.
Each object is designed for parallel access by multiple threads. Only a single block needs to be locked to update any value or key so multiple writes can occur at the same time. The data in each object is organized to find values or keys without inspecting every block. It is very fast and allows both OLTP and OLAP operations on the same data set.
So far, I have used them to attach tags to millions of files and do searches. I have created relational tables that I can query much faster than the same table in Postgres (https://www.youtube.com/watch?v=OVICKCkWMZE). I have created 3D relational tables that can be queried the same way 2D tables have traditionally been queried (https://www.youtube.com/watch?v=AXqvWMmoL1M).
The software is currently in an open beta and anyone can download and try it out on their own data set. https://www.Didgets.com
- Rolled a Linux distribution (PrintNanny OS) designed for mixing/matching controller software across a manufacturing "fleet".
- Created an atomic upgrade/rollback system.
- Built a WebRTC stack for live camera streaming.
- Lots of fun applying queue theory to distribute work across a fleet.
I'm learning a lot about how the government contract/bid process works, which I've always been insulated from as an engineer (even when working at companies known for gov support, like Red Hat).
My biggest customers tend to serve an ultra-specific niche, which is always fun to learn about. I love getting to talk to people who have spent thousands of hours cracking the laser tag gun market in Canada, or top RV parts supplier in Pacific Northwest, etc. There are entire worlds-within-worlds out there. =)
I have a Tech Center in Maryland (http://www.autechcenter.com) which offers lots of free services to Australian startups and we ticket clip any capital to make money.
Current projects are:
* NASA award winning Wound Healding tech https://rapair.com.au/
* AI for drug discovery https://www.polygenrx.com/
* 3D printing bone implants https://globalsurgicalinnovations.com.au/
* Water filtration - https://www.starwater.com.au/
It’s incredibly interesting and very niche. There are a handful of academic papers out there. I think it will be a big deal in the next 10 years.
The product seems simple, but there are so many opportunities for optimization. There's caching, CDNs, custom domain support, dynamically-generated open graph images, image optimizations, email sending, email reputations, analytics, and more.
There's this whole world of rules that accountants follow to properly record revenue. These rules are different for physical products, subscriptions, contracts, and even rules for what to do when you bundle these things together. And yet, the only two options for companies to crunch all this data is to use an Oracle product or Excel. (Or get the eng team to write SQL queries, to try and produce a correct result for a topic they don't have a background in!)
To do this, we're really leaning into databases like Clickhouse and DuckDB - taking raw data and very quickly summarizing them into journal entries that you can post during the month. Also using kinesis to let users stream in financial data in realtime, so we can then process that data in batch. (Kinesis -> S3 -> Lambda -> DB). In doing this, have also been writing low-level C and Go code to quickly iterate over a stream of transactions and accumulate results.
There are various other big dictionaries and websites specific to different dialects, but I don’t want to have to search on each of them by hand.
I’m currently working on a meta-dictionary, which is a fancy name for unique search engine for all the dictionaries I can find out there. It’s not finished but so far it works great; I use it almost daily.
I started with Phoenix (Elixir framework) and Svelte to learn them but I ditched Phoenix because it was a hell to work with how implicit things are there (no import; just assume that variable is defined "somewhere") and went back to a very basic Python Flask app that serves an API for Svelte.
Nothing very fancy on the tech side but it’s interesting to deep dive in the languages of Italy and learn a lot of things about both Italy and linguistics at the same time.
We are also getting ISO 27001 and we are going to open source the entire bundle of policies and the "information security management system" itself on GitHub, so I am also working on that.
I was pretty late to GME, despite having been on Reddit for a good few years. I started researching what had happened, and where the conversations were taking place just trying to understand where to place myself to be a part of the next one. I ended up checking out the typical r/WallStreeBets and the other big trading subreddits but I always felt that I'd need to devote 100% of my time to Reddit in order to capitalise on any of the information. So I built a scraper to do the heavy lifting for me. That worked well. Then I started trying to find ways to identify the good stocks from the bad. That worked even better. So I turned it loose onto many different social media platforms where people discuss trading.
It worked well enough that I ended up quitting my job to develop it and trade full time. There are still improvements that I'm working on but I think that'll always be the case.
A great proof of concept of this is BBBY last month. Out of nowhere it went from ~$5 to $30. There was no real business reason for this, just rumour and speculation. It won't be the last.
I don't know if links are allowed but if you want to follow along, it's https://feetr.io (unfortunately not accepting any more beta users currently, but all data is posted to Twitter until we launch). There's also a leaderboard at https://feetr.io/leaderboard if you're only interested in the big numbers.
The tech stack is 100% lisp. I use CCL locally, SBCL on AWS. Also, hunchentoot will power the backend when it launches. I mention that because a lot of people have this fear that hunchentoot is slow or something but in my case it's pretty rapid. Of course, make sure you're writing fast code, but it's not the bottleneck that I think people think it is.
Over the past couple of years, the popular search engines haven't given me great results, and I've had to use a series of different engines, search operators, and dorks to prod out the search results I'm looking for, with mixed success. The front ends have also become a bit too feature rich for my use cases; I prefer a minimal front end with just search results, but most front ends have links to news, videos, drop downs, cards, cookie banners, SEO spam, and have too much javascript. Too much time spent loading and rendering the page, and too many requests, in my opinion. Instead, my search engine only take 1 HTTP request per search result page and is only a few KB in size.
Right now this project is still in the early stages, and mostly proxies bing's api with some minor adjustments, but if anyone is interested in testing it out here is the link to the search engine [0] and a link to the blog with an RSS feed to follow future updates [1].
Open source, GitHub link at the bottom of the page. The API used serves a large part of Europe, so you might be able to fork the code and adapt it to another country with fairly minimal work.
Over a long time period, some perturbance befalls those cameras: gravity pulling them down, things knocking them, etc. That causes our accuracy to degrade over time, so we started looking at ways to continually heal that calibration without needing to do another stop-the-world calibration phase.
Turns out you can use the 3D objects themselves (actually the 2D constituent detections that you use to compute a 3D point) as correspondences instead of doing SIFT matching. The hard part is that you have to preserve the original coordinate frame after doing a bundle adjustment.
Computer vision is a ton of fun, and I highly recommend you give it a try if you’ve enjoyed data science stuff, but the incomprehensible “try it and see” engineering of working on deep neural nets turns you off.
I've been following the Tools for Thought crowd for a long while, and recently started looking into why we still have problems building interactive web apps, and where the issues and complexities arise. It's lead to examining the Clojure, Haskell, and Rust ecosystems, various papers, data structures, and databases. I've been writing my thoughts exploring the space at https://interjectedfuture.com
If you're looking for niche technologies, we've got a nice assortment at our podcast The Technium: https://www.youtube.com/channel/UCl_rEKDGBw4myn0uOnPxYsg
I started to teach myself unity in December 2021. I've personally experienced benefits to writing automated tests and using CICD; therefore, I thought it would be fun to learn about writing tests for a 3D based software. It will be different from the web/CLI based stuff I usually write.
So, it turns out that the video game community (or at least the online circles I frequent) are extremely against the idea of writing automated testing for various reasons. This translates into there effectively being a non-existent pedagogy around teaching how to write tests for 3D based software. Content is scarce, and the content you do find is produced by people who obviously don't write automated tests. So, for me I've hit the books to arbitrage & translate techniques and philosophies into this "untapped" domain.
It's interesting because I've figured out tips & tricks that I'd consider low hanging fruit... For example, if you place a "test" camera in the test case, you can actually see what's going on in the test when it's executed. Or how important it is to clean up every created game object in your test after each test; if done right, you can keep your SUT at origin (0,0,0). Or the importance of "test prefabs" who are effectively mocks of other "real" prefabs...
One innovation that I would like to use/build that I truly consider (((revolutionary))) is this: I want the test cases I write to also automatically (or when tagged with a certain C# attribute) generate the same game objects in a "exploratory(manual) test scene." I think this innovation is the "killer app" that will completely and totally sell the value proposition of automated tests to those opposed. If you decide to build this, please also publish it on openupm and reach out to me.
I have plenty of other thoughts and ideas on this space. I love talking about automated testing; it feels like a very futuristic programmer practice. I hope this post demonstrates why this work and space is so interesting.
I’m collaborating on an attempt to shift the responsibility for XSS from the developers and towards the browser. The current stage focuses on getting the use case "insert HTML but without any scripts" right.
There’s a public specification and prototype implementations in Firefox and Chrome. You can play with it here https://sanitizer-api.dev/
We think that we’ve made the best of many different tradeoffs, but we’re also keen to hear wide feedback.
Interspersed with that I have a huge amount of stuff to implement for my project still(search methods and the like), so I'm working on that as well. It's a lot of code to write and I'm just one guy so progress is slow.
What I'm doing at work would bore you to death so I'll leave it out.
It’s an absolute nightmare of paperwork and massively expensive fees and hardware. Communication systems and related fees may end up being more than 50% of the entire cost, it used to be over 75% but I gave up and decided to just make the entire thing bigger and more expensive because of how ludicrous it seemed to build a $10k cubesat and spend $135k just to get a permit to communicate with it!
It’s been a fascinating and rewarding endeavour but don’t get me started on how much I hate radio regulations and export control (ITAR) … I’ve had multiple business ideas along the way from all this learning but have done nothing about them because I quickly realised that I’m in the rare realm of needing a legal-expertise cofounder due to how insane these regulations are. I’ve honestly been told by some companies that telling me how much something costs to buy is both a trade secret requiring an NDA and that I’ll need to submit appropriate ITAR paperwork before they can even send me the NDA because if other people knew how much it cost that would be materially aiding efforts to reverse engineer such devices in circumvention of export controls… this is the insane world of aerospace engineering outside of academic projects and big defence subcontractors.
Still working on it though… it’s not like I’m in a rush, launch costs won’t come down for a while, likely 5 years once a couple more fully reusable launchers are competing with SpaceX’s Starship. Plenty of time to find solutions to these kinds of problems, like how I have to use software defined radio to build my own GPS unit since all commercial GPS units have speed and altitude lock outs to prevent use in guided missiles and military jets.
https://github.com/TimothyStiles/poly
Goal is to have a suite of packages and databases that can be used to design entirely novel proteins, metabolic pathways, and DNA constructs at scale because right now that software ecosystem just doesn't exist.
I also wrote a multithreaded actor implementation in Java. I am currently working on solving the expression problem and thinking of how entity component systems (ECS) can be used for parallelism and solving the problem of deciding how to map data to operations efficiently.
I am also thinking of how to patch running systems without downtime.
See my repository multiversion-concurrency-control for my parallel work https://github.com/samsquire/multiversion-concurrency-contro... and the userspace scheduler in preemptible-thread https://github.com/samsquire/preemptible-thread
See my profile for links to my journal/blog where I write of my progress everyday, specifically ideas4.
Based on my experiences, I think there is something missing in the market for content creators / editors for WebApps (with landing pages etc.) and websites. There are many headless CMSs available but many are lacking a simple editing experience.
I want to make it easy for the developer to integrate it into a project (using React for example) - but also make it really easy for the content creator to be more independent of developers - while keeping UI constraints etc.
I choose to build a native iOS and MacOS App with SwiftUI that basically provides all editing UI as native components instead of using HTML/CSS - Basically an overlay on top of the website that let's you control the components and website grid/layout. It's mainly a proof of concept and for me a way to learn SwiftUI :)
There are many Website builders that provide great editing UIs (e.g. Webflow, Builder.io etc.), but I feel a native app, especially on iPad OS could provide benefits and it's a nice challenge :)
Basically, you draw black and white images by spawning meatballs, then the drawing is rendered to a image and checked against a reference using openCV.
I found gesture algorithm online's but they always take into account individual strokes, and the drawing system is not using strokes.
I also have requirements to match the *whole* image but not *exactly*.
So what I'm doing is :
1. Resizing and cropping the input image to size of the reference image.
2. Matching each shape in the source image to the closest shape in the reference image (if any shape is below a certain threshold, the drawing is failed).
3. Blur the whole image a bit, and count the percentage of overlapping pixels (if over a certain value, this makes sure the different elements of the drawing are approximately in the right position).
This took me like 4 days to figure out, but it was really interesting having to prototype it using python and translate it to C# afterwards. (Though I sadly ended up buying a 80€ Unity OpenCV library since nothing else was working for me)
The natural-language plaintext is fed into a language model that annotates it. That output is converted into an intermediate representation based on relational algebra, which we manipulate and finally compile to SQL.
It's probably not scientifically groundbreaking, but there are a lot of interesting technical bits along the way, at least for me.
2. AI writing tutors for middle schoolers
3. Paper-digital integration to support equitable education in India
4. Systematic review of resonance in human interactions
5. Computational models of beauty
6. Generating new Socratic dialogues by fine-tuning GPT3 with the complete works of Plato
7. Promoting ADD as a valid lifestyle choice
I have been using a barebones version with a minimal and slightly UX hostile (for most peoples standards) version as a personal programming language. There was a random comment in a thread asking if I had anything readable for others. So I started getting a github based ‘blog’ together and started porting the compiler to JavaScript (which I don’t know very well) to have some interesting examples/demonstrations of the type theory and compilation process.
I don’t think the language will be released, but maybe some people will find the theoretical or implementation presentations helpful or interesting.
How I ended up working on this? There were a few ideas I wanted to work on. Most of them are much simpler and have easier path to monetization. But this kept coming back to the top of my thoughts stack, so I gave up fighting it. It ended up much harder than I even thought it was.
Niche tech? Explored F# Fable and ReScript. Both are really still too young so they added too much typing complexity to overcome, so it's just mainly JS and some typescript.
1. Make a list of tasks you need to do.
2. Take on one task and complete it.
3. Take On Me was a cool song.
4. Oh, I should go watch the video
...
I'm not doing very well at it.
Its basically a JavaScript DSL to create little web apps with reactive, understandable code. For example:
const age = input(“what is your age?”)
text(“your age is” + age)
Would automatically create a UI around this.
Can see it at https://trytoolkit.com
I’ve been surprised how much I’ve personally been getting value from this little app.
The interesting parts are convenience features like sharing an account across multiple devices, allowing temporary access to a device, etc.
The goal is to implement everything user side - it would act a bit like single sign on but with no need for any trusted third party. I’m starting with a browser plug-in.
My vision is a new client with an interpreter for BYOND's proprietary language that can execute SS13's existing content unchanged. Compile the client to WebAssembly so people can play and develop the game in a browser without installing any software. Unshackle the game from depending on hosted servers by making the client peer-to-peer using WebRTC. If the new client dethroned the BYOND client, then incompatible improvements could then be made to its fork of the BYOND language.
The name for this project in my head is "Babylon 13". :)
- Performance: How to scan files as fast as possible?
- Tag-Storage: How to store audio tags as generic as possible?
- File-Watchers: How to prevent fully indexing the filesystem over and over again and only react to changes?
- File-Sources: How to manage multiple file scanners as efficient as possible?
- Cancellation: How to cancel running tasks, when File-Sources are removed?
- Moved files: How to not lose all customized information (rating, playlists, playcount, etc.) when a file gets moved to another place?
- and many more :-)
It's probably not very interesting to other people, but I find it fascinating.
It's something a lot of folks struggle with yet it is a trait that is eminently learnable[2]!
[0]https://clay.earth [1]https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3498890/ [2]https://journals.sagepub.com/doi/abs/10.1177/014616721663784...
I suspect there will be a lot of companies in the near future that make AI models with permissive licenses easier to use, along different niches (like I am doing for video). It is a really promising area and picking a niche helps with papering over the challenges with general models.
It's a lot more technically challenging than we thought. For example, what's the best way to statistically guage a student's knowledge level? How do we recognize when an ngram counts as a new concept?
We haven't launched publically yet, but we have a LinkedIn page of you'd like to see when we do. https://www.linkedin.com/company/conceptionary/
The overall design ethos/constraints have posed some interesting challenges along the way, but so far I've had good luck developing with the following: Vue.js, Vuetify, vuex, Chart.js, Threads.js for orchestrating web workers, vue-router, a bunch of smaller libraries, Paddle, Firebase, and some Google Cloud Functions (the latter only come into play for those that upgrade + choose to enable cloud sync).
For now, it's still a side project :)
My current problem is I have concurrency working, but there is a aspect to movement that I have little information on, and the client isn't informative enough to help. So I am kinda stuck. I plan to do some manual testing of just fiddling with a value and see what happens. Its fun, but very demoralizing at the same time to see multiple people logged in, but their characters jumping around and not properly being moved around on other people's screens.
I also don't want to make any changes to client code, because that's our only source of truth to work with.
It's like oil; many sub-products can be made from this.
For starters, many of them:
- Currency exchange basic calculator.
- Commissions calculator (many exchanges claim to be "no commissions," but you pay a lot in exchange rate flexibility as well)[0]
- Some financial institutions need to track economic indicators... I have that data.
- Integration with CRMs, to issue correct quotes using parallel exchange rates.
- Integration with ERPs, for whatever you use ERPs. - etc. etc. etc.
EDIT: Clarity.
--
[0]: https://www.ivanmontilla.com/blog/how-do-zero-commissions-cu...
I started doing research on cameras around 2012 and have been obsessed every since. The company behind this code failed unfortunately but I've kept poking at it and finding more people interested in it, even as my day job is more about cloud infrastructure. Companies like Zippin which have carved out a niche in the cashierless space really impress me.
I am on the journey of learning computer science and programming on my own from awesome books and resources available out there.
I came across this article by Peter Norvig: https://norvig.com/21-days.html after I read his code of a spell checker: https://norvig.com/spell-correct.html. I was mesmerized by the code itself. It is so clean as if it is speaking to me. No series of spaghetti statements making me cringe just at the sight of it. I want to be able to achieve that skill. That is my project/problem if you can say so. In college I was studying electrical engineering when I was taught programming in C using a very bad book and I hated coding after that. Now, I am gradually beginning to see the beauty in it. It is just prose. Prose can be bad and excellent. I was taught bad prose that's why I dislike it.
I hope to become an elegant and useful programmer one day and build things that are useful to people.
If you have any particular resource in mind that can help me in my journey, don't forget to suggest some in the replies.
A few examples:
1. Dialtone using dual-tone multi-frequency signaling and 56K dial-up modem connection: https://www.youtube.com/watch?v=FomWraKuDFg&list=PLn67ccdhCs...
2. Deluxe Multitone Car Alarm: https://www.youtube.com/watch?v=A4uKcvZL7HM&list=PLn67ccdhCs...
3. Composition using only sounds from Windows 98 and XP https://www.youtube.com/watch?v=6lT-jr9sS6Y&list=PLn67ccdhCs...
4. Piano Music (Ballade Pour Adeline): https://www.youtube.com/watch?v=RnAfrEk429w&list=PLn67ccdhCs...
5. Electronic Music Demo: https://www.youtube.com/watch?v=MllJLIX1glg&list=PLn67ccdhCs...
- A tool for resolving external executables and sourceables in Shell scripts to absolute paths (and erroring when they aren't found). This is mainly to make it possible to package Shell with Nix in a way that they don't depend on all of their dependencies being in the environment.
The top level of this is ~easy enough, but identifying executables in the arguments to other commands is significantly harder. So, much of the focus is on triaging executables to flag those most likely to exec, coming up with human/automated processes for triaging the source of those executables to confirm, finding a good way to express complex nonstandard CLI syntax for parsing (in a way that's humane enough to be maintainable and ~contributable), etc.
- A toolchain for single-sourcing documentation in order to generate idiomatic markup for multiple output languages with different whitespace semantics. Because it's really hard to get good idiomatic format conversions out of largely presentational source such as markdown, this includes a flexible language with DIY semantics.
The entire toolchain does something a bit like what you'd try to accomplish with j2cli, some yaml, some jinja templates (and probably some python plugins, if you're trying to do anything complex) with more precise whitespace control. Still very rough/early, but glad I finally bit the bullet and started on it.
It's challenging and interesting work, and I'd recommend building your own as a way to gain deeper understanding of math.
The good thing about RL is that it is adaptive and proactive - it tunes control from data as it streams and it learns to act with delayed feedback in mind.
The bad thing is that it can be sample inefficient. It needs lots and lots of data. That is not good in a time-critical case such as when faults occur. I've come to enjoy pushing back on completely black-box control of systems, foregoing the advances of decades of theory. That helps me scale back complexity of the system if only a little domain knowledge is sprinkled on the solution.
So, first, I started using transfer learning to re-use existing policies from similar situations. However "similar" is a very open problem in the domain. I then thought that we can exploit insights we get from classical control of linear systems into these more complex domains. If, for example, we can make some simplifying assumptions (locally linear, there's some global optimum etc.). So far I am getting promising results. But there is always some data wrangling involved. I am hoping to use that in my dissertation later this year. (If anyone's hiring, feel free to reach out).
I spent the last year or so exploring and validating solutions to the "voice cliff" problem found in multimodal voice-controlled devices such as smart displays and TVs. That's when you are controlling some UI with your voice and suddenly you can't, because that particular screen hasn't been designed for voice. Think of Echo Show settings or literally all of Nest Hub's UI. As a result, I came up with a "big idea". So now I want to file some preliminary patent applications for some of the bits that might actually be new inventions. The chances of actually getting a patent are slim - most of the ideas I thought were brilliant were already thought up, patented, published in research papers or even productized, sometimes a decade ago or more! There are a lot of forward thinking researchers out there. And this doesn't even take into account patents which are pending. Apple could have a tranche of patents announced tomorrow which were submitted years ago. Or some startup could come out of stealth and you learn that a well funded group of people way smarter than you has been working on similar problems for years.
Anyways, actual new ideas are rare, but I seem to have a few.
Distilling a big idea into smaller pieces that can be explained in detail is a good exercise, regardless of the eventual success of any patents. So I'm churning through 100+ pages of notes, twice as many patent and paper references, and a bit of code and trying to get it into a state to share with the world. I've never been an academic, but I used to work at Nokia Research Center, so I've worked with quite a few and now I know how hard their job was.
https://text-generator.io/blog/text-generator-now-researches...
It also does embeddings for text/images and code. The problem i suppose is general agi where you have unstructured input and unstructured output and you need to do a lot of research in the process of producing the output which one day may include training, using other web services/information retrieval etc.
Looking to expand the system to understand more and more unstructured inputs and be able to output more and more advanced outputs like images, audio, video, datasets etc.
I was using OpenAI a lot and realised it's going to be dang expensive which is why I started https://text-generator.io so more people can use it
I needed a privacy respecting notes app which worked like whatsapp. So I built exactly that - https://play.google.com/store/apps/details?id=com.makenoteto...
Took inspiration from Signal and called it Note to Self.
I'm also building a new, high performance, high quality 2D rendering engine that runs primarily on the GPU.
And I'm also exploring new approaches to building GUI, primarily in Rust. A new focus is to integrate that more tightly with the renderer mentioned above. A benefit I expect is that I'll be able to do the CPU side of the rendering work in multiple threads, where existing architectures make it very difficult to break out of a single thread.
How did I end up working on this? It's a natural evolution of what I've been working on for a long time, in Ghostscript, on Google Fonts, working on the text stack on Android, and now fortunate enough to have a research position back on Google Fonts where I am able to focus on this and have great support from the rest of the team.
* There aren't many
* Those that you can find, don't support Typescript, so it's all guesswork.
* Don't have much (if any) support for related data queries
So I've been working on a ORM for PostgreSQL and Typescript for a few months now in my spare time [0]. It's been probably one of the biggest, challenging, and interesting problems I have embarked on solving.
Some things i've observed along the road:
* Typescript is weird and wonderful when you push it to it's limits, particularly around recursive structures. Strange workarounds exist, nicely provded by all of the two Stack Overflow Typescript gurus that exist.
* PostgreSQL doesn't have some of the features that you would expect from other DB technologies (i.e. certain table alterations, limit on update and delete, etc.). Weird and wonderful workarounds exist, but it always leaves you thinking _why aren't those workarounds just the de facto way for doing those things?_.
It's a multiplayer sandbox (no rules enforcement) that just allows you to play by providing all the different zones the game involves (play area, hand, lib, gy, ex) and letting you free-drag between them. It has features for every mechanic the game involves - tapping, turning face-down, switching card control to another player, etc - and it looks good and is smooth and easy to use.
There's nothing like it. Existing options have various constraints/drawbacks or else are so old-fashioned and clumsy as to be nigh-unusable.
I don't know how people will react when I announce. I don't know how Wizards of the Coast will react. I'm a bit nervous about it. I know it's the best of its class, because I play magic, and I have no idea what other people will make of it. It fits their fair-use policy (product must be free, but can be ad-supported or take patreon-revenue) but even so I'm nervous.
A few experiments planned around
- design of heat absorbers and heat sinks
- night and day operation using radiative cooling/paints & coatings
As a security guy previously, particularly on the offensive side, this has been a fascinating endeavor thus far.
I’m particularly interested in: the application of PRP injection to various ailments in the body (where else are there oceans of stem cells we can activate to help the body heal itself), and the application of ML to surfacing data about ailments for populations that are woefully understudied (trans patients undergoing GAHT, for example).
- Traditional SIEM tools are not a good fit for large amounts of data — they're either too expensive or come with a high ops burden. - Your data is locked into vendor specific formats.
We solve this by ingesting and storing all your data in Apache Iceberg tables on S3, (allowing for low cost queries directly on S3 data + an open table format) and building everything using only serverless technologies(Lambda, SQS, S3, Athena, etc.).
We also let you write realtime Python detections as code and are written in Rust, which makes for some interesting technical challenges.
It began as a toy/experiment that turned out to be useful, so I decided to pour the gas on and push it to it's natural conclusion.
It's close to hitting a first alpha release at which point I expect it to stabilize somewhat.
More info : https://scallyw4g.github.io/poof/
The website's a WIP, but the tool does compile to WASM so you can give it a try in the browser, which is pretty neat-o.
I'm also actively looking for feedback. If anyone has constructive comments/criticism feel free to reach out :)
To put it on scifi-like phrases : "The universal knowledge repository of humankind". Or "imagine if we can extract the entirety of knowledge that stored inside a certain person brain, then store it in the machine. So everyone else could access that knowledge."
Wikipedia, OpenStreetMap, social media platform, search engine and the Internet come close, but still there are rooms for improvement. Especially in "discovery" part and "overall structure" within a body of knowledge. Sure, there are quality knowledge content, but they are mostly spread out and undiscoverable and very unstructured.
So this time around I'm slowly asking people I know to try it out. Honestly it takes a little courage and has given me a newfound respect for people who do sales (though it's a little different, as the whole thing was my idea). There does seem to be a big difference between an explicit ask to sign up vs. hey check this out. Oftentimes people will say that's so great but not sign up (which I try not to take personally).
Also on perfect hashes: https://github.com/rurban/nbperf, also gperf and cmph improvements.
Try it live here (hit "view sample data"): https://studio.foxglove.dev/
And it's open source! https://github.com/foxglove/studio
Shameless plug - we're hiring: https://foxglove.dev/careers
For fun, I just implemented a finite-state machine for a videogame I'm working on in Rust. It's used to swap out sprite animations depending on the state of its parent entity. I'm glad to be solving problems I'd never come across in my normal line of work.
This is not just "proof of existence" by inserting hashes of content into a Merkle tree, like Haber & Stornetta implemented in 1990 and like we talked about on the Cypherpunks email list in the early 1990s, although of course that would be part of a complete system.
Nor is it relying on just the metadata that e.g. TruePic uses. I have been developing additional methods to asymptotically approach provability that the content depicted actually happened in real life. Think undetectable watermarks by multiple observers (courts want to see multiple witnesses) but it goes beyond that.
Relevant to this are efforts by CAI, C2PA, and companies like TruePic. I am hoping my methods are different enough, with little enough overlap, that we can realize them without licensing their IP. My goal is a more decentralized system that protects the privacy of witnesses.
My contact info is in my profile. I'd love to discuss this with anyone. I'd even be relieved if someone would explain that my ideas are not useful, or already have been done, or even would have unintended negative consequences for society. (It's hard to stay current in the relevant literature on signal processing, steganography, cryptography, and zero-knowledge proof, plus politics and game theory.)
I would even considering simply donating my trade secrets (such as they are) to some person or organization who impresses me as sincere and likely to move them forward, to avoid having to sit on them hands-off (again) during my tenure at (yet another) company that is about to offer me a full-time job today to work on something unrelated. While some of my ideas "toward authenticatability" may have been ahead of their time when I first wrote them up ten years ago, they probably aren't anymore, so the window is (at best) closing now.
Moving from largely centralized and specific message passing and token bridging into general interoperability that is decentralized and modular for any new blockchain or multi-chain protocol to extend and build on top of is a huge challenge.
This involves talking with people about their experiences and writing about what I've learned from them and observed from my own sabbatical.
If anyone is curious, I have some posts up at https://www.albertwavering.com/tags/sabbatical/ and would love to chat with anyone who is thinking about or currently taking a break.
Ps : I knew today's Standard technology was 360°photos. But I was convinced this was not an immersive enough technology.
Immersion, for me, should feel more like a video game FPS.
So I gave it a try: - A No-Coding Editor, just photos as input, plus 20 minutes of clicks and you have your immersion.
Solo project
Most podcast feeds only include last ~100 episodes. Podcasts such as Planet Money and Radio Lab have more than 1000 episodes in total.
Manually combining the feeds is too time consuming and I enjoy building this tool.
So I took a few steps back, getting deeper and deeper into Stoicism, and brainstorming new Stoic-based business models in the e-commerce space, or Art, or both!
I spent 8 years as a cancer scientist in drug discovery space, till I realised it doesn't make much sense to keep searching for more potential targets, if they will end up on a shelf for lack of clinical trials budget.
I am trying to solve this issue by platforming the infrastructure. Bringing web2.0 to clinical drug development.
the current site texts you when shows matching your preferences are about to start. the next iteration will have visualizations of the music being played by each station in close to real time and allow you to select between them, save them, etc. the visualizations will be updated each minute and colors will be selected by the HSL color system, where hue = tempo (from blue for slower to red for faster), saturation (or intensity of color) = loudness, and lightness = pitch (technically, spectral centroid or 'center of mass' of frequencies of music).
i also want the ui to allow people to select from stickers i've taken pictures of around nyc and arrange them similar to concert venue bathrooms or middle school lockers haha.
altogether, want to create a fun way to explore independent radio!
We are building a cybersecurity marketplace for professionals looking for remote assignments. Currently it is in beta version and we are looking for test users to check functionality. We will provide gift credits for our testers when we are live so that they can be featured sellers on the main page. Just let me know if you are interested :)
Also, most investment sites are deeply textual while investment information is more effective visually.
I am having another go at it, this time with Swift Charts and SwiftUI on the Mac.
We have some pretty big clients using this in anger, and they’re wild about it.
It’s not glamorous work, but it’s important!
So much educational content is locked behind videos or hard to search mediums like an example project you have to download and setup.
There are attempts at community wiki but I'm trying to start an opiniated blog/wiki type site since I dont agree with lots of other things that I wish existed.
Why?
Because creating these websites quickly is always content problem. We should start from content and create a design according to content not other way around.
A HTTP request is a function call with named arguments. What if the server had some endpoints that provide basic fp (map, fold, curry, …), and you let your users create their own endpoints by currying together other endpoints?
Synthetic aperture radar scraper and aggregator across international public satellite providers.
And reading statistics in my free time.
Was trying to replicate methods from some papers and couldn’t get it to work. Seemed promising but alas. Now I’m just recovering from all that jazz.
I am a former principal engineer at a FAANG/MAMAA/etc. [8] company turned independent educational researcher, and I believe I have developed a novel edtech approach to address this learning gap. I am collaborating with a few data scientists and educators, but am searching for UX designers, curriculum designers, writers, and marketers who can help make this a reality.
In brief, one of the core challenges in edtech is how to build an engaging app that actually results in learning (many apps resort to psychological tricks [9] and gamification hacks [10] that are long-term detrimental.) Although I have a high-level direction for how to achieve this, my skill set is mostly in backend development and public speaking.
If you are interested, please reach out to (my HN username) @ (Microsoft's not cold email domain acquired in 1997).
[0] https://www.nationsreportcard.gov/
[1] https://www.nationsreportcard.gov/highlights/ltt/2022/
[2] https://caldercenter.org/sites/default/files/CALDER%20Workin...
[3] https://en.wikipedia.org/wiki/Follow_Through_(project)
[4] https://www.nifdi.org/what-is-di/project-follow-through
[5] https://www.researchgate.net/profile/Russell-Gersten/publica...
[6] https://files.eric.ed.gov/fulltext/ED472569.pdf
[7] http://arthurreadingworkshop.com/wp-content/uploads/2018/05/...
[8] MANGA, GAMMA, NAAAM, M2A3, Big Tech, Big 4, Big 5, or really whatever you would like to call them.
[9] Skinner boxes, daily rewards, resource decay, loss aversion, etc.
[10] Leaderboards, points, badges, etc.
This is a very difficult problem, to trade successfully based on rules. But it's also very interesting.
I’m a contractor, I work through my company. It’s way more tax efficient than being perm.
But, it could be even more efficient.
So, I did some research. Turn out, where I am, a business can have multiple business codes. Activities the company is registered as performing. Now, I was and still am a software development company. But I also added artistic production as a business code.
Why artistic productions? Well, because it’s the most liberal business code for expenses.
As a software company, I can expense travel for some things, the most useful for me being conferences. But that means I need to buy the conference ticket even if I’m not interested and can only expense accommodations for the duration of the conference. That’s good but a bit restrictive.
Now, as a artistic production company, I’m not limited by that. As long as I can prove I traveled in order to produce, well, just about anything that can be construed as being art, I can expense travel and I can expense accommodations for as long as I need. You can’t rush art, you know how it is.
So, obviously, first thing I did, I whipped up a blog, posted some random pictures every day and called it good. I’m not using company money to travel the world you see, I’m using company money in order to produce art. Haven’t paid a euro of my own money on accommodation or travel for close to a year now. And the best part, I get to deduce this from my profit.
I love the tax system. I really do.
But, doing this, I got curious. I thought, hey, why not make it better. So I thought let’s see what all the rage is with dalle and stable diffusion and all that. So, what I’m doing now is, I’m changing my approach. My art will no longer be just pictures taken quickly with my iphone. No no. I’m getting the picture and I’m sending it to an API I built. The API runs some computer vision, labelling what it sees, some classification, some stuff with the pic metadata, tries to put it all together and turns it into a description of the picture.
The description of the picture I took is used as a prompt for stable diffusion. And that result ends up being the art. I’d like to thank of my art as a modern commentary on the age old “is art imitating life or is life imitating art?” - At least that’s what I told my accountant and he concerned, it would stand up in the case of an audit!
So, yeah, tldr, I’m working on image to text and text to image to expense my nomad lifestyle on the company.
How could it be changed in a substantial, positive, intentional way, ideally in a surprisingly short time frame?
The dirty secret about the restaurant industry is that most independent restaurants are very close to bankruptcy and the people running it often do not know that until it's too late.
There was a thread a few days ago about the true cost of making hardware devices often being less than 30% of the retail price. From a business perspective, the same should apply to every item on your menu as well. Except more often than not, the person running the restaurant often has no idea how much a meal costs to serve.
The real issue is that people start restaurants not because they want to run a business, but because they like food. In a way it is a good thing that they don't approach it with a capitalist mindset, as I don't thing we'd have the same variety of food if everyone did, but more often than not they are running blind and just hoping for the best. They put in long hours and a lot of their own capital, and if they are lucky they can survive a few years by being able to pay their staff, suppliers and rent, and finally being able to afford to pay themselves enough to personally survive on.
Generate a planar graph, preferably without a Delaunay triangulation.
One was an indoor web map built on an entirely open-source stack (QGIS and OpenLayers): https://map.fieldmuseum.org/ It was an interesting challenge because the open-source web mapping libraries (and many/most of the commercial ones too) were designed for outdoor use and didn't really account for indoor idiosyncrasies. Take rooms and doors, for example... while you can import vector SVGs geoJSONs to these mapping libraries, you can't easily indicate "this shape is solid except for these two doors" -- whether for visuals or for an eventual turn-by-turn routing/navigation layer. Another challenge was the concept of "floors", overlapped groups of geometry (walls, balconies, whatever) and points of interest that are mutually exclusive. We ended up having to trace floors against each other in QGIS (map editing software) and then adding state at the Javascript level to store each floor as a separate object to render or hide. Then there were a bunch of UX/UI tweaks we had to do, like how to draw arrows for Covid one-way flows, how zooming should work (we categorize POIs according to level of importance, then show/hide them at different zooms, and also dynamically scale fonts to ensure readability), which geometries to make clickable (a sidebar opens with pics and more details), how to hide easter eggs.
The whole thing was written as a vanilla JS single-page app tied to a headless CMS (Contentful at first, eventually DatoCMS) so that editors could easily change copy, graphics, etc. But editing geometries (the shapes and positions of things) still required QGIS knowledge or at least the ability to manually edit geoJSON files.
We launched this as a MVP with the intent of rapidly following up with additional features (blue-dot positioning, turn-by-turn routing, audio directions, a better codebase, etc.) At the end of the project we ended up open-sourcing it (after much begging and pleading with The Powers That Be), but then soon thereafter abandoned it altogether :( Just as well, really, because the code was really terrible -- I can say that with confidence because I wrote it, lol. But it's still an interesting problem space. As far as I know, there isn't a ready-built solution for this sort of stuff... especially not an open-source/source-available one. Some commercial solutions have limited indoor support, but as of our last evaluation (early 2021), none of them were especially powerful, elegant, or user-friendly. Hope that keeps evolving!
-----------
Fast-forward to my current job, and we're now working on a similarly graphical frontend app. I now work for a solar company, and we're designing a map-like tool to help installers plot out the PV modules (the technical term for a single "solar panel") on their customers' roofs, which then enables historical monitoring of various metrics like power output, temperature, etc.
Aside from the graphical challenges (rendering React states to Canvas shapes), there were also some interesting frontend engineering challenges, like how to plot tens of thousands of points in a chart while allowing real-time scrubbing in a timeline, but where no two datapoints were ever at the same time, because the devices we were working with all communicated in a one-at-a-time queue.
Anyway, I won't get too much into the details here... nothing that exciting, just a lot of nuance. What we're building is like a super-simplified version of https://www.opensolar.com/3ddesign (which is awesome! check it out).
---------
But overall, I just love that I get to work on these sorts of fun apps instead of bog-standard blog pages or ecommerce sites. Even a few years ago, things like this would've been complex desktop apps with difficult lifecycles to manage... now they can be web apps that are updated multiple times a week and evergreen everywhere, to all our customers. Pretty cool stuff!
I feel very lucky to be in my line of work -- web dev for nonprofits and small/medium businesses in interesting spaces, who have challenges unique to their industries, not just "how do we optimize this database" or the such (though there is that too). It was easy enough to pick up these technologies and frameworks in just a few months or years. I'm very grateful that is a career option for people now... I'm a high school dropout who eventually went to a natural resources school, but couldn't find a job in that industry so picked this up instead. It's all entirely self-teachable (with tutorials and Stack, of course).
It is my way of providing ammunition for those who uphold humanism against the technocrats who think they can automate testing.
- a GPS receiver
- a 9-DOF magnetometer/accelerometer/gyroscope board
- temperature sensors
- microphones
- cameras
- etc
and a processor, which can sense the physical world, and - possibly - facilitate learning in a manner somewhat akin to the way an infantile human learns. All of this is rooted in my belief that while embodiment may not be strictly necessary for AGI, it is probably very useful. And the reason I believe that is because I believe an awful lot of our early development is experiential and rooted in experiencing the physical world. That is, we develop our naive / intuitive understanding of physics ("there is up/down and if I drop something it falls") and our naive metaphysics and epistemology of the world ("there are objects and objects persist even when I can't see them briefly", etc) through what we see and hear and feel. So I want to try to simulate that aspect somewhat, but without necessarily going "full robot". That is, my current hardware platform can sense, but it isn't ambulatory and it doesn't currently have any ability to manipulate or affect anything in the physical world, except by generating noise (speech) and possibly by blinking LED's.
Anyway, the GPS stuff is all setup and now I'm working on code for the accelerometer board. I'm using one of those Adafruit BNO055 boards for that. Currently working on getting the sensor readings off of it using I²C.
At present the model has individual code modules for the various sensor types, and new sensor events go to two places:
- sent via MQTT to persistent storage on a server. This is to facilitate pos-hoc analysis, off-line training of ML models, and possibly replay scenarios.
- turned into a percept and sent to a "percept queue" using ActiveMQ. This all happens on the processor (a Raspberry Pi) embedded in the hardware platform.
Then code modules listen on the percept queue for incoming percepts and then react. The initial idea is to use a very SOAR like "cognitive loop" as the primary "thing" that listens and responds to the percepts. But that will probably change over time.
Very likely it will eventually turn into something inspired by a combination of things:
- the old blackboard architecture approach
- Minsky's "society of mind" stuff
- the BDI (Beliefs-Desires-Intentions) model
- etc.
Not to mention, of course, elements of very contemporary approaches to Machine Learning - Deep Neural Networks, Reinforcement Learning, etc.
When I get to the vision and audio stuff it gets more interesting because there will need to be more initial processing of the audio and video streams, and more front-end intelligence to decide what counts as a "percept". I mean, you wouldn't send every video frame or every audio sample. So now you get into figuring out what kind of delta counts as a "change" that's worth of generating a percept and possibly attracting the attention of the system. And now you get into the multi-modal part, because think about how human interactions work. If you want someone's attention you may approach them and say "excuse me", OR you might approach, say "excuse me" AND tap them on the shoulder, etc. So attention might involve physical touch, sound, vision, etc. all combined. So lots of interesting stuff to explore around how to deal with all of that.
An interesting aside to this is that I get to give Bosworth senses that humans don't have. For example, with the GPS board, Bosworth can "know" its location on the surface of the Earth to a fairly reasonable level of precision, at all times, even if blindfolded and transported elsewhere. Likewise it will know the precise time at all tims. And Bosworth will also "know" which direction magnetic north is at all times, and its velocity, etc. Now, will having those abilities be useful, or will this just confuse the issue? Who knows? But exploring that is part of what makes this fun.