Hello folks, we've found a performance bottleneck searching for items in a large in-memory collection. Most of the code was converted from 15+ year old C++ code into C# so a lot of weird looking stuff came across. There are lots of sequential searches for items in arrays and List<> which have complexity O(n) and get really slow for large collections. I previously converted one bottleneck List<> into a Dictionary<> and it was a miracle cure, but it was a simple case with one property value being a unique key. The next bottleneck collection needs to be searched efficiently on different property values, and I'm wondering if anyone knows of a collection which supports multiple indexes (possibly non-unique). An in-memory SQLite table with multiple indexes would do the job, but it's a bit heavyweight and like hammering a screw. Using multiple Dictionaries or Sets to index the same objects won't work due to non-unique values. Maybe there is some lightweight package around for this sort of thing? It's a classic computer science problem. *Greg Keogh*
Hi Greg, Long time no chat... What I have done is have a list of objects and had multiple dictionaries indexing it. An object that is in a dozen data structures only exists in memory once, so it is cheap to setup the extra indexes. You can get some great performance from it. As soon as you put a database in there, you have to get the data in and out of the database, that can be very time consuming depending on what you are doing. It sounds like you are going to need an dictionary of lists, to solve the duplicate non-unique key problem. All the best Greg H Greg Harris Sydney Australia On Sat, Sep 28, 2024 at 9:47 AM Greg Keogh via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Hello folks, we've found a performance bottleneck searching for items in a large in-memory collection. Most of the code was converted from 15+ year old C++ code into C# so a lot of weird looking stuff came across. There are lots of sequential searches for items in arrays and List<> which have complexity O(n) and get really slow for large collections. I previously converted one bottleneck List<> into a Dictionary<> and it was a miracle cure, but it was a simple case with one property value being a unique key.
The next bottleneck collection needs to be searched efficiently on different property values, and I'm wondering if anyone knows of a collection which supports multiple indexes (possibly non-unique).
An in-memory SQLite table with multiple indexes would do the job, but it's a bit heavyweight and like hammering a screw. Using multiple Dictionaries or Sets to index the same objects won't work due to non-unique values. Maybe there is some lightweight package around for this sort of thing? It's a classic computer science problem.
*Greg Keogh* -- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
Hi Greg, Long time no chat... What I have done is have a list of objects and had multiple dictionaries indexing it. An object that is in a dozen data structures only exists in memory once, so it is cheap to setup the extra indexes. You can get some great performance from it. As soon as you put a database in there, you have to get the data in and out of the database, that can be very time consuming depending on what you are doing. It sounds like you are going to need an dictionary of lists, to solve the duplicate non-unique key problem. site: https://ezmusic.ir/
Hi Greg, As Arman suggested, it could literally be as simple as Dictionary<(key1,key2,key3),List<value>), although creating a list for each entry when duplicate keys are uncommon might be a waste. A second dictionary which only contains the value not in a list Dictionary<(key1,key2,key3),value), could be used initially then fall back to the list dictionary as needed. Wrap it all up in an abstraction to hide the complexity. On Wed, 6 Aug 2025 at 02:14, Arman AZ via ozdotnet <ozdotnet@ozdotnet.com> wrote:
Hi Greg,
Long time no chat...
What I have done is have a list of objects and had multiple dictionaries indexing it. An object that is in a dozen data structures only exists in memory once, so it is cheap to setup the extra indexes. You can get some great performance from it. As soon as you put a database in there, you have to get the data in and out of the database, that can be very time consuming depending on what you are doing.
It sounds like you are going to need an dictionary of lists, to solve the duplicate non-unique key problem.
site: https://ezmusic.ir/ -- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- Alan Ingleby
What is going on here??? Original request from Greg Keogh: 28-Sept-2024 One and only response from me (Greg Harris) the following day: 29-Sept-2024 Then today, effectively an exact duplicate of my response 6 hours ago with my signature panel removed, line endings changed, plus a reference added to a random website???? I expect this is some form of attempt to get exposure for the random web site???? Image of diff (using https://www.diffchecker.com/text-compare/)... [image: image.png] Greg Harris Harris Consulting Group Pty Ltd Greg@HarrisConsultingGroup.com www.HarrisConsultingGroup.com <http://harrisconsultinggroup.com> phone: (international) +61 407 942 982 phone: (within Australia) 0407 942 982 Sydney Australia On Wed, Aug 6, 2025 at 2:12 AM Arman AZ via ozdotnet <ozdotnet@ozdotnet.com> wrote:
Hi Greg,
Long time no chat...
What I have done is have a list of objects and had multiple dictionaries indexing it. An object that is in a dozen data structures only exists in memory once, so it is cheap to setup the extra indexes. You can get some great performance from it. As soon as you put a database in there, you have to get the data in and out of the database, that can be very time consuming depending on what you are doing.
It sounds like you are going to need an dictionary of lists, to solve the duplicate non-unique key problem.
site: https://ezmusic.ir/ -- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
Oh I hadn't even noticed that! Yeah looks like a Spam? Weird. On Wed, 6 Aug 2025 at 09:16, Greg Harris via ozdotnet <ozdotnet@ozdotnet.com> wrote:
What is going on here??? Original request from Greg Keogh: 28-Sept-2024 One and only response from me (Greg Harris) the following day: 29-Sept-2024 Then today, effectively an exact duplicate of my response 6 hours ago with my signature panel removed, line endings changed, plus a reference added to a random website????
I expect this is some form of attempt to get exposure for the random web site????
Image of diff (using https://www.diffchecker.com/text-compare/)...
[image: image.png]
Greg Harris Harris Consulting Group Pty Ltd Greg@HarrisConsultingGroup.com www.HarrisConsultingGroup.com <http://harrisconsultinggroup.com> phone: (international) +61 407 942 982 phone: (within Australia) 0407 942 982 Sydney Australia
On Wed, Aug 6, 2025 at 2:12 AM Arman AZ via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Hi Greg,
Long time no chat...
What I have done is have a list of objects and had multiple dictionaries indexing it. An object that is in a dozen data structures only exists in memory once, so it is cheap to setup the extra indexes. You can get some great performance from it. As soon as you put a database in there, you have to get the data in and out of the database, that can be very time consuming depending on what you are doing.
It sounds like you are going to need an dictionary of lists, to solve the duplicate non-unique key problem.
site: https://ezmusic.ir/ -- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- Alan Ingleby
Yeah user joined 2 days ago apparently. I've removed them from the lsit. On Wed, 6 Aug 2025 at 09:59, Alan Ingleby via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Oh I hadn't even noticed that! Yeah looks like a Spam? Weird.
On Wed, 6 Aug 2025 at 09:16, Greg Harris via ozdotnet < ozdotnet@ozdotnet.com> wrote:
What is going on here??? Original request from Greg Keogh: 28-Sept-2024 One and only response from me (Greg Harris) the following day: 29-Sept-2024 Then today, effectively an exact duplicate of my response 6 hours ago with my signature panel removed, line endings changed, plus a reference added to a random website????
I expect this is some form of attempt to get exposure for the random web site????
Image of diff (using https://www.diffchecker.com/text-compare/)...
[image: image.png]
Greg Harris Harris Consulting Group Pty Ltd Greg@HarrisConsultingGroup.com www.HarrisConsultingGroup.com <http://harrisconsultinggroup.com> phone: (international) +61 407 942 982 phone: (within Australia) 0407 942 982 Sydney Australia
On Wed, Aug 6, 2025 at 2:12 AM Arman AZ via ozdotnet < ozdotnet@ozdotnet.com> wrote:
Hi Greg,
Long time no chat...
What I have done is have a list of objects and had multiple dictionaries indexing it. An object that is in a dozen data structures only exists in memory once, so it is cheap to setup the extra indexes. You can get some great performance from it. As soon as you put a database in there, you have to get the data in and out of the database, that can be very time consuming depending on what you are doing.
It sounds like you are going to need an dictionary of lists, to solve the duplicate non-unique key problem.
site: https://ezmusic.ir/ -- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
-- Alan Ingleby -- ozdotnet mailing list To manage your subscription, access archives: https://codify.mailman3.com/
participants (6)
-
Alan Ingleby -
Arman AZ -
David Connors -
Greg Harris -
Greg Keogh -
pakeen＠iinet.net.au