First steps you take when designing a library/framework?

Or more specifically a piece of application that is long-lasting or modular that aids you or specific community of other programmers in solving a technological problem when writing their business logic.

Do you first start tinkering with the implementation or use case? Do you begin drafting data structs, interfaces first or maybe you just go with the flow?