Hi Tom et al, Coincidentally, my CoPilot trial ends today and I've decided to let it continue at $US10/month. So my AI qualifications now begin with one month of GitHub Copilot. Is this particular AI useful for developers? Urrrr...sort of...I've got mixed feelings, but I'm going to let it run for some more months and see how it goes. People in demos everywhere, even many of the top boffins have it running in their videos, so if it's good enough for Mads and Anders then I like what they like. Sometimes Copilot throws out suggestions that are jaw-droppingly psychically predictive and I stare in disbelief before hitting tab. Other times it produces something completely irrelevant that was snipped from a different project I was previously working on. The former seems to outnumber the latter, so that's why I'm continuing the subscription. As Tom hints, I also probably get a 1-2% boost in productivity thanks to Copilot (well, I will when my reflexes adapt to stop having keyboard races with it). I feel the name "AI" is a bit generous, as it's clearly just trawling through vast amounts of code and samples, analysing it and spitting fragments back out at you based on patterns and context. There are fabulous algorithms at work finding patterns, but is this "AI"? When I first saw the movie *2001: A Space Odyssey* in 1969 I was fascinated by HAL (I was clearly destined to work in IT!), and now 55 years later I'd like to be able to sit down with HAL and explain some complex business requirement to him, converse like professionals and weigh-up the pros and cons of different platforms and languages based upon his world of experience and get him to code like a 1000x developer and generate a complete working skeleton of the required product. When will I see that? I'm a bit disappointed so far. Oh well, back to coding with my Copilot buddy. *Greg K* On Sun, 28 Jul 2024 at 23:19, Tom Gao via ozdotnet <ozdotnet@ozdotnet.com> wrote:
Hey Guys, it's been awhile since I've asked the question about the usefulness of AI in our development world - for a seasoned senior developer for a complex digital transformation project. Since then I've met the microsoft board member on OpenAI and have completed an AI course at Stanford and seen the latest and greatest at the most advanced world leading AI lab. To be honest I think this AI thing is going to be 10 years away to be truly useful as it is hyped to be. But this is just a personal opinion.
I wanted to get some feedback from you guys last time michael said maybe about 1-2% in improvement in usefulness to a seasoned dev. I've had ongoing robust conversations. Because I feel the benefits are completely overstated in general (open to being corrected). The most senior tech lead in my team believe it's a negative benefit. As in it can't understand complex business analyst requirements, nor debug codes, nor find / correct defects. The effort to prompt engineer and also to correct what's generated is also understated. Plenty of tools already do what chatgpt/co-pilot can do. In this case we've experimented with github copilot which claims to provide 55% efficiency in coding.
What's everyone's thoughts since Feb this year? - how useful is it in real world dev tasks, building complex transformation projects from BA requirements?
Thanks in advance, Tom
On Sat, Feb 24, 2024 at 9:21 PM Michael Ridland via ozdotnet < ozdotnet@ozdotnet.com> wrote:
If you’re a seasoned practitioner in the language it’s probably sitting around 1-2%, because it can help with some code conversions. Copilot code completions are amateur. For .net resharper/rider etc is actually a much better option than copilot, practical use of that is probably around 15% productivity increase.
Copilot does help if you don’t know the language or technology, which maybe commbank staff are not skilled up enough? ;)
On Sat, 24 Feb 2024 at 6:27 pm, DotNet Dude via ozdotnet < ozdotnet@ozdotnet.com> wrote:
I’m not a subscriber to read the entire article but 30% is a load of BS at this stage
On Sat, 24 Feb 2024 at 15:18, Tom Gao via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Here's the article.
https://www.afr.com/technology/commbank-claims-ai-is-already-making-it-work-...
So what does everyone think about the claim that 30% efficiency? ie to me as head of IT instead of hiring 10 developers to build applications now only 7 is required. To me that just doesn't seem realistic.
On Fri, Feb 23, 2024 at 3:04 PM Tom Rutter via ozdotnet < ozdotnet@ozdotnet.com> wrote:
On Fri, 23 Feb 2024 at 12:32, Dr Greg Low via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Worse, in the baseball umpiring exam I mentioned, someone said to me “isn’t that what you’d get if you just asked a whole lot of fans about the rules rather than asking umpires?”
There’s probably something important about that. How does it know which of the material it was trained on is valid?
Yep that’s the thing, it doesn’t actually know what is “valid”. From what I understand it just gets “pushed” in certain directions during training and not in all cases obviously.
My experience with things like ChatGPT so far basically brings it down to a better Google really. Early days
Regards,
Greg
Dr Greg Low
1300SQLSQL (1300 775 775) office | +61 419201410 mobile
SQL Down Under | Web: https://sqldownunder.com <https://urldefense.proofpoint.com/v2/url?u=https-3A__sqldownunder.com_&d=DwM...> | About Greg: https://about.me/greg.low <https://urldefense.proofpoint.com/v2/url?u=https-3A__about.me_greg.low&d=DwM...>
*From:* mike smith via ozdotnet <ozdotnet@ozdotnet.com>
*Sent:* Friday, February 23, 2024 12:19 PM *To:* ozDotNet <ozdotnet@ozdotnet.com> *Cc:* mike smith <meski.oz@gmail.com> *Subject:* Re: AI
"old system views"
That makes me wonder if it has any way of differentiating between something it found from a decade ago to more recent data.
Mike
On Fri, 23 Feb 2024, 11:43 Dr Greg Low via ozdotnet, < ozdotnet@ozdotnet.com> wrote:
Hi Tom,
For me, it depends what you want it to do. It certainly can appear to help someone who’s new to an area.
For most code writing, I’ve been pretty underwhelmed. As an example, if I ask it to write SQL, I get a very poor outcome. It will use old deprecated views instead of the current system views (that have been around for a decade), and often does things in a convoluted way.
What I have been impressed with, is how it can help you understand acronyms, etc. Quite amazing. I’ve also been pretty impressed with using it go generate some test data, including in multiple languages. And the test data is fairly believable. If I ask it for family names, and I also ask for Chinese, it does pick common Chinese family names in the test output. That’s pretty impressive.
It can do a reasonable job of things like “here’s some DAX code, can you simplify it?” It often can. Or “here’s a regular expression, can you explain what it does?” and it does that just fine. I’ve seen people happily using it to explain code that they don’t understand, or to (sort of) document some code.
But it also is so confident on things, yet so wrong. I gave it a 25 question baseball umpire test the other day. It was 100% confident sounding, but 40% correct. The weird thing is that some of the questions that it got right, are things that new human umpires often get wrong. Yet for simpler questions, it would say that something legal is illegal.
It’s certainly interesting, but it’s very much a work in progress. It will be part of our futures.
Regards,
Greg
Dr Greg Low
1300SQLSQL (1300 775 775) office | +61 419201410 mobile
SQL Down Under | Web: https://sqldownunder.com <https://urldefense.proofpoint.com/v2/url?u=https-3A__sqldownunder.com_&d=DwM...> | About Greg: https://about.me/greg.low <https://urldefense.proofpoint.com/v2/url?u=https-3A__about.me_greg.low&d=DwM...>
*From:* Tom Gao via ozdotnet <ozdotnet@ozdotnet.com> *Sent:* Friday, February 23, 2024 11:58 AM *To:* ozDotNet <ozdotnet@ozdotnet.com> *Cc:* Tom Gao <tom@tomgao.com> *Subject:* AI
Hi guys, I haven't posted in a few years and haven't been on the tools for a long time now as well. I'm on a panel on a digital conference coming up in march. We had a pre meeting today and the topic of AI came up. Two of the panelist said cited CBA and Westpac using AI and were able to save 30% on development effort.
Personally I just finished an AI course my view is quite the opposite. My personal opinion of the generative AI space and AI in general having spent time with the academics is that the benefits are significantly over inflated.
I want to get some other opinions if you are seeing any significant benefit and that I may be just out of touch or not aware.
Thanks,
Tom
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
Copilot definitely isn't a magic 'do my job' button, and likely won't ever be, or that job would simply no longer exist. It does take care of a lot of the busy work though, and isn't just for Intellisense like code suggestions - you can happily converse with it in the chat window about high level topics and refine the context incrementally, generate boilerplate json/bicep/scripts from a few lines of prose, or highlight a non-trivial block of existing code and ask it to explain/summarise or find issues. While it's not HAL 9000 I reckon it comes pretty close, and quite importantly hasn't tried to murder me (yet). Before my trial I was skeptical about how much value it would provide, but was pleasantly surprised enough to hand over 15 dollarydoos each month. 😎 cheers, Tony On 30/07/2024 08:06, Greg Keogh via ozdotnet wrote:
When I first saw the movie /2001: A Space Odyssey/ in 1969 I was fascinated by HAL (I was clearly destined to work in IT!), and now 55 years later I'd like to be able to sit down with HAL and explain some complex business requirement to him, converse like professionals and weigh-up the pros and cons of different platforms and languages based upon his world of experience and get him to code like a 1000x developer and generate a complete working skeleton of the required product. When will I see that? I'm a bit disappointed so far.
Oh well, back to coding with my Copilot buddy.
/Greg K/
Thanks Tony GitHub claims 55% efficiency for devs gained do you agree? If not what % would you give it for day to day dev work? We’ve given it a 1-2% at the most. On Tue, 30 Jul 2024 at 3:54 PM, Tony McGee via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Copilot definitely isn't a magic 'do my job' button, and likely won't ever be, or that job would simply no longer exist.
It does take care of a lot of the busy work though, and isn't just for Intellisense like code suggestions - you can happily converse with it in the chat window about high level topics and refine the context incrementally, generate boilerplate json/bicep/scripts from a few lines of prose, or highlight a non-trivial block of existing code and ask it to explain/summarise or find issues.
While it's not HAL 9000 I reckon it comes pretty close, and quite importantly hasn't tried to murder me (yet).
Before my trial I was skeptical about how much value it would provide, but was pleasantly surprised enough to hand over 15 dollarydoos each month. 😎
cheers, Tony
On 30/07/2024 08:06, Greg Keogh via ozdotnet wrote:
When I first saw the movie *2001: A Space Odyssey* in 1969 I was fascinated by HAL (I was clearly destined to work in IT!), and now 55 years later I'd like to be able to sit down with HAL and explain some complex business requirement to him, converse like professionals and weigh-up the pros and cons of different platforms and languages based upon his world of experience and get him to code like a 1000x developer and generate a complete working skeleton of the required product. When will I see that? I'm a bit disappointed so far.
Oh well, back to coding with my Copilot buddy.
*Greg K*
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
That 55% seems to be from here? https://github.blog/news-insights/research/research-quantifying-github-copil... They're comparing two cohorts on how long it took to complete a well defined task with Copilot vs without, so yeah with the automatic suggestions that it produces, that feels like a somewhat plausible number for a specific coding task. They're not claiming a 55% efficiency gain across general overall monthly productivity though, that'd be a ridiculously high. Overall productivity boost is tough to quantify because tasks aren't always well defined, and nobody is coding every single minute of the day, but even tiny boost may be enough for a subscription to pay for itself. 🙂 cheers, Tony On 30/07/2024 16:18, Tom Gao wrote:
Thanks Tony GitHub claims 55% efficiency for devs gained do you agree? If not what % would you give it for day to day dev work?
We’ve given it a 1-2% at the most.
On Tue, 30 Jul 2024 at 3:54 PM, Tony McGee via ozdotnet <ozdotnet@ozdotnet.com> wrote:
Copilot definitely isn't a magic 'do my job' button, and likely won't ever be, or that job would simply no longer exist.
It does take care of a lot of the busy work though, and isn't just for Intellisense like code suggestions - you can happily converse with it in the chat window about high level topics and refine the context incrementally, generate boilerplate json/bicep/scripts from a few lines of prose, or highlight a non-trivial block of existing code and ask it to explain/summarise or find issues.
While it's not HAL 9000 I reckon it comes pretty close, and quite importantly hasn't tried to murder me (yet).
Before my trial I was skeptical about how much value it would provide, but was pleasantly surprised enough to hand over 15 dollarydoos each month. 😎
cheers, Tony
On 30/07/2024 08:06, Greg Keogh via ozdotnet wrote:
When I first saw the movie /2001: A Space Odyssey/ in 1969 I was fascinated by HAL (I was clearly destined to work in IT!), and now 55 years later I'd like to be able to sit down with HAL and explain some complex business requirement to him, converse like professionals and weigh-up the pros and cons of different platforms and languages based upon his world of experience and get him to code like a 1000x developer and generate a complete working skeleton of the required product. When will I see that? I'm a bit disappointed so far.
Oh well, back to coding with my Copilot buddy.
/Greg K/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
My Copilot trial is expiring today, and I'm still wondering whether I should continue it. It's ... ok. Most of the time it saves me some keystrokes. I needed a Levenshtein Distance function the other day and it was decent - only required a small tidy up. Only a few times have I been wondering what it's thinking. Much better than the early days with ChatGPT. One annoying thing since installing it is when dealing with long method signatures, intellisense is no longer telling me which parameter I'm up to. Probably something in the configuration somewhere for that. I was listening to a PodCast about the future of Copilot, and I think it's going to get far more interesting (useful) in the future. They recognize that coding is a small minority of what we do. I know some people say they spend 40% coding, but for myself it's probably closer to 10%. What they're working towards (i.e. are currently dog-fooding themselves), is being able to give it requirements. It will look through the code-base, and create "slots" with suggestions on how it would implement it, which you can accept/reject or change. I'm currently working on a legacy ERP system with many years worth of "bandaids" (most of which were well before my time). And understanding the code-base takes more time than making the change itself. So this would actually move the needle on overall productivity. Nathan. On Tue, 30 Jul 2024 at 13:57, Tony McGee via ozdotnet <ozdotnet@ozdotnet.com> wrote:
Copilot definitely isn't a magic 'do my job' button, and likely won't ever be, or that job would simply no longer exist.
It does take care of a lot of the busy work though, and isn't just for Intellisense like code suggestions - you can happily converse with it in the chat window about high level topics and refine the context incrementally, generate boilerplate json/bicep/scripts from a few lines of prose, or highlight a non-trivial block of existing code and ask it to explain/summarise or find issues.
While it's not HAL 9000 I reckon it comes pretty close, and quite importantly hasn't tried to murder me (yet).
Before my trial I was skeptical about how much value it would provide, but was pleasantly surprised enough to hand over 15 dollarydoos each month. 😎
cheers, Tony
On 30/07/2024 08:06, Greg Keogh via ozdotnet wrote:
When I first saw the movie *2001: A Space Odyssey* in 1969 I was fascinated by HAL (I was clearly destined to work in IT!), and now 55 years later I'd like to be able to sit down with HAL and explain some complex business requirement to him, converse like professionals and weigh-up the pros and cons of different platforms and languages based upon his world of experience and get him to code like a 1000x developer and generate a complete working skeleton of the required product. When will I see that? I'm a bit disappointed so far.
Oh well, back to coding with my Copilot buddy.
*Greg K*
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
I find I use copilot.microsoft.com far more than GitHub copilot. I find I have better "conversations" with the Microsoft site and it can handle and critique any code I put in too On Wed, 31 July 2024, 3:14 pm Nathan Schultz via ozdotnet, < ozdotnet@ozdotnet.com> wrote:
My Copilot trial is expiring today, and I'm still wondering whether I should continue it. It's ... ok. Most of the time it saves me some keystrokes. I needed a Levenshtein Distance function the other day and it was decent - only required a small tidy up. Only a few times have I been wondering what it's thinking. Much better than the early days with ChatGPT. One annoying thing since installing it is when dealing with long method signatures, intellisense is no longer telling me which parameter I'm up to. Probably something in the configuration somewhere for that.
I was listening to a PodCast about the future of Copilot, and I think it's going to get far more interesting (useful) in the future. They recognize that coding is a small minority of what we do. I know some people say they spend 40% coding, but for myself it's probably closer to 10%. What they're working towards (i.e. are currently dog-fooding themselves), is being able to give it requirements. It will look through the code-base, and create "slots" with suggestions on how it would implement it, which you can accept/reject or change. I'm currently working on a legacy ERP system with many years worth of "bandaids" (most of which were well before my time). And understanding the code-base takes more time than making the change itself. So this would actually move the needle on overall productivity.
Nathan.
On Tue, 30 Jul 2024 at 13:57, Tony McGee via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Copilot definitely isn't a magic 'do my job' button, and likely won't ever be, or that job would simply no longer exist.
It does take care of a lot of the busy work though, and isn't just for Intellisense like code suggestions - you can happily converse with it in the chat window about high level topics and refine the context incrementally, generate boilerplate json/bicep/scripts from a few lines of prose, or highlight a non-trivial block of existing code and ask it to explain/summarise or find issues.
While it's not HAL 9000 I reckon it comes pretty close, and quite importantly hasn't tried to murder me (yet).
Before my trial I was skeptical about how much value it would provide, but was pleasantly surprised enough to hand over 15 dollarydoos each month. 😎
cheers, Tony
On 30/07/2024 08:06, Greg Keogh via ozdotnet wrote:
When I first saw the movie *2001: A Space Odyssey* in 1969 I was fascinated by HAL (I was clearly destined to work in IT!), and now 55 years later I'd like to be able to sit down with HAL and explain some complex business requirement to him, converse like professionals and weigh-up the pros and cons of different platforms and languages based upon his world of experience and get him to code like a 1000x developer and generate a complete working skeleton of the required product. When will I see that? I'm a bit disappointed so far.
Oh well, back to coding with my Copilot buddy.
*Greg K*
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
participants (5)
-
Greg Keogh -
Nathan Schultz -
Tom Gao -
Tony McGee -
Tony Wright