I wanted to get some feedback from you guys last time michael said maybe about 1-2% in improvement in usefulness to a seasoned dev. I've had ongoing robust conversations. Because I feel the benefits are completely overstated in general (open to being corrected).
I don't think the benefit is overrated and we see huge value in the use of AI across all of the dev work we do. I think you're possibly looking at it glass half empty instead of glass half full. The question is not whether the AI is smarter than you, can write better code than you - the question is whether you get a great productivity boost in day to day activities.
Co-pilot smashes boring drudge work like making constructors out of the park. You can use ChatGPT to break the back of time consuming but easy work - for example, upload a CSV and then ask it to write you the DDL for a table to store that and a stored procedure to do CRUD operations etc. It churns it out instantly.
If you asked me to write an UPSERT statement in T-SQL or create a user in MySQL I would have to Google it. They're tasks that are easy, but also tasks that I seldom do so don't have the syntax front of mind. I can just leave out the middle-man and get the AI to do it for me.
The most senior tech lead in my team believe it's a negative benefit. As in it can't understand complex business analyst requirements, nor debug codes, nor find / correct defects.
All of our PRs are reviewed by AI before humans touch them (via Code Rabbit). People reviewing PRs are generally more senior, more expensive and time poor. Code Rabbit produces *really* good summaries of what is in the PR and the implications. It provides a lot of low level commentary that is useful to the dev's themselves before asking for a review. You can talk back and forth with Code Rabbit in the comments under the PR in GitHub and ask for suggestions etc.
The effort to prompt engineer and also to correct what's generated is also understated.
What sort of things are you asking it to do? If you're trying to get an LLM to write conceptually complex things then that might be where you have friction.
Here's an example of a CodeRabbit review for a simple change done today in one of our systems together with example of dev conversation. The reviews of complex changes (I could not find a good example without internal information) are astonishingly good and the back and forth with the bot can be very nuanced (arguing the point on why there isn't actually a race condition in some code because it isn't considering this or that). This level of review / suggestion / fix cycle happens before a senior considers the PR.
The poems are a nice touch :)
Conversation on that change with CodeRabbit: