This paper proposes a novel high throughput internet protocol (IP) lookup engine, which is built upon a recently proposed multiple pipeline array architecture that has parallel two-dimensional circular search capabilities on intersecting and variable length pipelines. Our new engine is composed of specially designed processing elements (PEs) including dual input/output static random access memory units and bidirectional links, hence allowing search to proceed in all directions and admitting search requests from all PEs at the boundary of the array. We propose a novel data structure called compact clustered trie (CCT), which is better than traditional binary trie in terms of memory requirement and number of memory accesses. We develop novel approaches including a CCT forwarding table construction method, a mapping strategy and a suitable IP lookup algorithm. Our new lookup engine achieves a much higher average case throughput and a much lower average delay compared with existing IP lookup solutions making, for example, an 8 Tbps high-speed router front end possible. The engine is also well suited for the IPv6 addressing scheme.